Image processing device and image processing method

ABSTRACT

An image processing device including an acquiring section configured to acquire quantization matrix parameters from an encoded stream in which the quantization matrix parameters defining a quantization matrix are set within a parameter set which is different from a sequence parameter set and a picture parameter set, a setting section configured to set, based on the quantization matrix parameters acquired by the acquiring section, a quantization matrix which is used when inversely quantizing data decoded from the encoded stream, and an inverse quantization section configured to inversely quantize the data decoded from the encoded stream using the quantization matrix set by the setting section.

CROSS REFERENCE TO RELATED APPLICATION

The present application is a continuation of U.S. application Ser. No.15/955,091 filed Apr. 17, 2018, which is a continuation of U.S.application Ser. No. 14/536,851 filed on Nov. 10, 2014 (granted as U.S.Pat. No. 9,986,241 on May 29, 2018), which is a continuation of U.S.application Ser. No. 13/990,489 filed on May 30, 2013 (granted as U.S.Pat. No. 9,706,205 on Jul. 11, 2017), which is the National Stage ofInternational Application No. PCT/JP2012/050931 filed on Jan. 18, 2012,and which claimed priority to Japanese Application Nos. 2011-027896filed on Feb. 10, 2011, 2011-047655 filed on Mar. 4, 2011, and2011-187179 filed on Aug. 30, 2011. Each of the above-listed documentsis hereby incorporated by reference in their entirety.

TECHNICAL FIELD

The present disclosure relates to an image processing device and animage processing method.

BACKGROUND ART

In H.264/AVC, one of the standard specifications for image encodingschemes, it is possible to use different quantization steps for eachcomponent of the orthogonal transform coefficients when quantizing imagedata in the High Profile or higher profile. A quantization step (orquantization scale) for each component of the orthogonal transformcoefficients may be set on the basis of a quantization matrix (alsocalled a scaling list) defined at the same size as the units oforthogonal transform, and a standard step value.

FIG. 38 illustrates four classes of default quantization matrices whichare predefined in H.264/AVC. The matrix SL1 is the default 4×4quantization matrix for intra prediction mode. The matrix SL2 is thedefault 4×4 quantization matrix for inter prediction mode. The matrixSL3 is the default 8×8 quantization matrix for intra prediction mode.The matrix SL4 is the default 8×8 quantization matrix for interprediction mode. The user may also define one's own quantization matrixthat differs from the default matrices illustrated in FIG. 38 in thesequence parameter set or the picture parameter set. Note that in thecase where no quantization matrix is specified, a flat quantizationmatrix having an equal quantization step for all components may be used.

In High Efficiency Video Coding (HEVC), whose standardization is beingadvanced as a next-generation image encoding scheme to succeedH.264/AVC, there is introduced the concept of a coding unit (CU), whichcorresponds to a macroblock of the past (see Non-Patent Literature 1below). The range of coding unit sizes is specified in the sequenceparameter set as a pair of power-of-2 values called the largest codingunit (LCU) and the smallest coding unit (SCU). Additionally, SPLIT_FLAGsare used to designate a specific coding unit size within the rangespecified by the LCU and the SCU.

In HEVC, one coding unit may be split into one or more units oforthogonal transform, or in other words, one or more transform units(TUs). Any of 4×4, 8×8, 16×16, and 32×32 is usable as the transform unitsize. Consequently, a quantization matrix may also be specified for eachof these candidate transform unit sizes. Non-Patent Literature 2 belowproposes specifying multiple quantization matrix candidates for onetransform unit size in one picture, and adaptively selecting aquantization matrix for each block from the perspective ofrate-distortion (RD) optimization.

CITATION LIST Non-Patent Literature

-   Non-Patent Literature 1: JCTVC-B205, “Test Model under    Consideration”, Joint Collaborative Team on Video Coding (JCT-VC) of    ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11 2nd Meeting: Geneva, CH,    21-28 Jul. 2010-   Non-Patent Literature 2: VCEG-AD06, “Adaptive Quantization Matrix    Selection on KTA Software”, ITU—Telecommunications Standardization    Sector STUDY GROUP 16 Question 6 Video Coding Experts Group (VCEG)    30th Meeting: Hangzhou, China, 23-24 Oct. 2006

SUMMARY OF INVENTION Technical Problem

However, the quantization matrix adapted to quantization and inversequantization differs according to the characteristics of each imageincluded in a video. For this reason, the frequency of quantizationmatrix updates will rise if one attempts to encode video whose imagecharacteristics change from moment to moment with optimal quantizationmatrices. In H.264/AVC, the quantization matrix is defined in thesequence parameter set (SPS) or the picture parameter set (PPS).Consequently, if the frequency of quantization matrix updates rises, theproportion of the encoded stream occupied by the SPS or the PPS willincrease. This means that the encoding efficiency will decrease becauseof increased overhead. For HEVC, in which the quantization matrix sizeis further increased and in which several different quantizationmatrices may be defined for each picture, there is a risk that suchdecreases in the encoding efficiency accompanying the update of thequantization matrix may become even more significant.

Consequently, it is desirable to provide a mechanism enabling moderationof the decrease in encoding efficiency accompanying the update of thequantization matrix.

Solution to Problem

According to an embodiment of the present disclosure, there is providedan image processing device including an acquiring section configured toacquire quantization matrix parameters from an encoded stream in whichthe quantization matrix parameters defining a quantization matrix areset within a parameter set which is different from a sequence parameterset and a picture parameter set, a setting section configured to set,based on the quantization matrix parameters acquired by the acquiringsection, a quantization matrix which is used when inversely quantizingdata decoded from the encoded stream, and an inverse quantizationsection configured to inversely quantize the data decoded from theencoded stream using the quantization matrix set by the setting section.

The image processing device may be typically realized as an imagedecoding device that decodes an image.

According to an embodiment of the present disclosure, there is providedan image processing method including acquiring quantization matrixparameters from an encoded stream in which the quantization matrixparameters defining a quantization matrix are set within a parameter setwhich is different from a sequence parameter set and a picture parameterset, setting, based on the acquired quantization matrix parameters, aquantization matrix which is used when inversely quantizing data decodedfrom the encoded stream, and inversely quantizing the data decoded fromthe encoded stream using the set quantization matrix.

According to an embodiment of the present disclosure, there is providedan image processing device including a quantization section configuredto quantize data using a quantization matrix, a setting sectionconfigured to set quantization matrix parameters that define aquantization matrix to be used when the quantization section quantizesthe data, and an encoding section configured to encode the quantizationmatrix parameters set by the setting section within a parameter setwhich is different from a sequence parameter set and a picture parameterset.

The image processing device may be typically realized as an imageencoding device that encodes an image.

According to an embodiment of the present disclosure, there is providedan image processing method including quantizing data using aquantization matrix, setting quantization matrix parameters that definethe quantization matrix to be used when quantizing the data, andencoding the set quantization matrix parameters within a parameter setwhich is different from a sequence parameter set and a picture parameterset.

Advantageous Effects of Invention

According to an image processing device and an image processing methodin accordance with the present disclosure, it is possible to moderatethe decrease in encoding efficiency accompanying the update of thequantization matrix.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating an exemplary configuration of animage encoding device according to an embodiment.

FIG. 2 is a block diagram illustrating an example of a detailedconfiguration of the syntax processing section illustrated in FIG. 1.

FIG. 3 is an explanatory diagram illustrating exemplary parametersincluded in a quantization matrix parameter set in an embodiment.

FIG. 4 is an explanatory diagram illustrating exemplary parametersincluded in a slice header in an embodiment.

FIG. 5 is a flowchart illustrating an exemplary flow of a parameter setinsertion process according to an embodiment.

FIG. 6 is an explanatory diagram for explaining differences in thestream structure between a technique according to an embodiment and anexisting technique.

FIG. 7 is a block diagram illustrating an exemplary configuration of animage decoding device according to an embodiment.

FIG. 8 is a block diagram illustrating an example of a detailedconfiguration of the syntax processing section illustrated in FIG. 7.

FIG. 9 is a flowchart illustrating an exemplary flow of a quantizationmatrix generation process according to an embodiment.

FIG. 10 is a flowchart illustrating an example of a detailed flow of aprocess in copy mode according to an embodiment.

FIG. 11 is a flowchart illustrating an example of a detailed flow of aprocess in axis designation mode according to an embodiment.

FIG. 12 is a flowchart illustrating an exemplary flow of a process forsetting a quantization matrix to a slice according to an embodiment.

FIG. 13 is a first explanatory diagram illustrating a first example ofillustrative pseudo-code expressing the syntax of a quantization matrixparameter set.

FIG. 14 is a second explanatory diagram illustrating a first example ofillustrative pseudo-code expressing the syntax of a quantization matrixparameter set.

FIG. 15 is a third explanatory diagram illustrating a first example ofillustrative pseudo-code expressing the syntax of a quantization matrixparameter set.

FIG. 16 is a first explanatory diagram illustrating a second example ofillustrative pseudo-code expressing the syntax of a quantization matrixparameter set.

FIG. 17 is a second explanatory diagram illustrating a second example ofillustrative pseudo-code expressing the syntax of a quantization matrixparameter set.

FIG. 18 is a third explanatory diagram illustrating a second example ofillustrative pseudo-code expressing the syntax of a quantization matrixparameter set.

FIG. 19 is a fourth explanatory diagram illustrating a second example ofillustrative pseudo-code expressing the syntax of a quantization matrixparameter set.

FIG. 20 is a fifth explanatory diagram illustrating a second example ofillustrative pseudo-code expressing the syntax of a quantization matrixparameter set.

FIG. 21 is an explanatory diagram illustrating an example ofquantization scale setting areas defined in order to quantize aquantization matrix.

FIG. 22 is an explanatory diagram illustrating an example ofquantization scales set in the respective quantization scale settingareas illustrated by example in FIG. 21.

FIG. 23 is an explanatory diagram for explaining 11 classes of VLCtables prepared in LCEC.

FIG. 24 is an explanatory diagram illustrating an example of an encodedstream structured in accordance with a first technique that uses an APS.

FIG. 25 is an explanatory diagram illustrating an example of APS syntaxdefined in accordance with a first technique that uses an APS.

FIG. 26 is an explanatory diagram illustrating an example of sliceheader syntax defined in accordance with a first technique that uses anAPS.

FIG. 27 is an explanatory diagram illustrating an example of APS syntaxdefined in accordance with an exemplary modification of a firsttechnique that uses an APS.

FIG. 28 is an explanatory diagram illustrating an example of an encodedstream structured in accordance with a second technique that uses anAPS.

FIG. 29 is an explanatory diagram illustrating an example of an encodedstream structured in accordance with a third technique that uses an APS.

FIG. 30 is an explanatory diagram illustrating an example of APS syntaxdefined in accordance with a third technique that uses an APS.

FIG. 31 is an explanatory diagram illustrating an example of sliceheader syntax defined in accordance with a third technique that uses anAPS.

FIG. 32 is a table listing parameter features for each of severaltypical encoding tools.

FIG. 33 is an explanatory diagram for explaining an example of anencoded stream structured in accordance with an exemplary modificationof a third technique that uses an APS.

FIG. 34 is a block diagram illustrating an example of a schematicconfiguration of a television.

FIG. 35 is a block diagram illustrating an example of a schematicconfiguration of a mobile phone.

FIG. 36 is a block diagram illustrating an example of a schematicconfiguration of a recording and playback device.

FIG. 37 is a block diagram illustrating an example of a schematicconfiguration of an imaging device.

FIG. 38 is an explanatory diagram illustrating default quantizationmatrices which are predefined in H.264/AVC.

DESCRIPTION OF EMBODIMENTS

Hereinafter, preferred embodiments of the present invention will bedescribed in detail with reference to the appended drawings. Note that,in this specification and the drawings, elements that have substantiallythe same function and structure are denoted with the same referencesigns, and repeated explanation is omitted.

Also, the description will proceed in the following order.

1. Exemplary configuration of image encoding device according toembodiment

-   -   1-1. Exemplary overall configuration    -   1-2. Exemplary configuration of syntax processing section    -   1-3. Exemplary parameter structure

2. Process flow during encoding according to embodiment

3. Exemplary configuration of image decoding device according toembodiment

-   -   3-1. Exemplary overall configuration    -   3-2. Exemplary configuration of syntax processing section

4. Process flow during decoding according to embodiment

-   -   4-1. Generating quantization matrices    -   4-2. Setting quantization matrix to slice

5. Syntax examples

-   -   5-1. First example    -   5-2. Second example

6. Various exemplary configurations of parameter sets

-   -   6-1. First technique    -   6-2. Exemplary modification of first technique    -   6-3. Second technique    -   6-4. Third technique    -   6-5. Exemplary modification of third technique

7. Applications

8. Conclusion

1. EXEMPLARY CONFIGURATION OF IMAGE ENCODING DEVICE ACCORDING TOEMBODIMENT

This section describes an exemplary configuration of an image encodingdevice according to an embodiment.

1-1. Exemplary Overall Configuration

FIG. 1 is a block diagram illustrating an exemplary configuration of animage encoding device 10 according to an embodiment. Referring to FIG.1, the image encoding device 10 is equipped with an analog-to-digital(A/D) conversion section 11, a reordering buffer 12, a syntax processingsection 13, a subtraction section 14, an orthogonal transform section15, a quantization section 16, a lossless encoding section 17, anaccumulation buffer 18, a rate control section 19, an inversequantization section 21, an inverse orthogonal transform section 22, anaddition section 23, a deblocking filter 24, frame memory 25, a selector26, an intra prediction section 30, a motion estimation section 40, anda mode selecting section 50.

The A/D conversion section 11 converts an image signal input in ananalog format into image data in a digital format, and outputs asequence of digital image data to the reordering buffer 12.

The reordering buffer 12 reorders the images included in the sequence ofimage data input from the A/D conversion section 11. After reorderingthe images according to a group of pictures (GOP) structure inaccordance with the encoding process, the reordering buffer 12 outputsthe reordered image data to the syntax processing section 13.

The image data output from the reordering buffer 12 to the syntaxprocessing section 13 is mapped to a bitstream in units called NetworkAbstraction Layer (NAL) units. The stream of image data includes one ormore sequences. The leading picture in a sequence is called theinstantaneous decoding refresh (IDR) picture. Each sequence includes oneor more pictures, and each picture further includes one or more slices.In H.264/AVC and HEVC, these slices are the basic units of videoencoding and decoding. The data for each slice is recognized as a VideoCoding Layer (VCL) NAL unit.

The syntax processing section 13 sequentially recognizes the NAL unitsin the stream of image data input from the reordering buffer 12, andinserts non-VCL NAL units storing header information into the stream.The non-VCL NAL units that the syntax processing section 13 inserts intothe stream include sequence parameter sets (SPSs) and picture parametersets (PPSs). Furthermore, in the present embodiment, the syntaxprocessing section 13 inserts into the stream a quantization matrixparameter set (QMPS), a non-VCL NAL unit different from the SPS and thePPS. The syntax processing section 13 also adds a slice header (SH) atthe beginning of the slices. The syntax processing section 13 thenoutputs the stream of image data including VCL NAL units and non-V CLNAL units to the subtraction section 14, the intra prediction section30, and the motion estimation section 40. A detailed configuration ofthe syntax processing section 13 will be further described later.

The subtraction section 14 is supplied with the image data input fromthe syntax processing section 13, and predicted image data selected bythe mode selecting section 50 described later. The subtraction section14 calculates prediction error data, which is the difference between theimage data input from the syntax processing section 13 and the predictedimage data input from the mode selecting section 50, and outputs thecalculated prediction error data to the orthogonal transform section 15.

The orthogonal transform section 15 performs an orthogonal transform onthe prediction error data input from the subtraction section 13. Theorthogonal transform executed by the orthogonal transform section 15 maybe discrete cosine transform (DCT) or the Karhunen-Loeve transform, forexample. The orthogonal transform section 15 outputs transformcoefficient data acquired by the orthogonal transform process to thequantization section 16.

The quantization section 16 uses a quantization matrix to quantize thetransform coefficient data input from the orthogonal transform section15, and outputs the quantized transform coefficient data (hereinafterreferred to as quantized data) to the lossless encoding section 17 andthe inverse quantization section 21. The bit rate of the quantized datais controlled on the basis of a rate control signal from the ratecontrol section 19. The quantization matrix used by the quantizationsection 16 is defined in the quantization matrix parameter set, and maybe specified in the slice header for each slice. In the case where aquantization matrix is not specified, a flat quantization matrix havingan equal quantization step for all components is used.

The lossless encoding section 17 generates an encoded stream byperforming a lossless encoding process on the quantized data input fromthe quantization section 16. The lossless encoding by the losslessencoding section 17 may be variable-length coding or arithmetic coding,for example. Furthermore, the lossless encoding section 17 multiplexesinformation about intra prediction or information about inter predictioninput from the mode selecting section 50 into the header of the encodedstream. The lossless encoding section 17 then outputs the encoded streamthus generated to the accumulation buffer 18.

The accumulation buffer 18 uses a storage medium such as semiconductormemory to temporarily buffer the encoded stream input from the losslessencoding section 17. The accumulation buffer 18 then outputs the encodedstream thus buffered to a transmission section not illustrated (such asa communication interface or a connection interface with peripheralequipment, for example), at a rate according to the bandwidth of thetransmission channel.

The rate control section 19 monitors the free space in the accumulationbuffer 18. Then, the rate control section 19 generates a rate controlsignal according to the free space in the accumulation buffer 18, andoutputs the generated rate control signal to the quantization section16. For example, when there is not much free space in the accumulationbuffer 18, the rate control section 19 generates a rate control signalfor lowering the bit rate of the quantized data. Also, when there issufficient free space in the accumulation buffer 18, for example, therate control section 19 generates a rate control signal for raising thebit rate of the quantized data.

The inverse quantization section 21 uses a quantization matrix toperform an inverse quantization process on the quantized data input fromthe quantization section 16. The inverse quantization section 21 thenoutputs transform coefficient data acquired by the inverse quantizationprocess to the inverse orthogonal transform section 22.

The inverse orthogonal transform section 22 performs an inverseorthogonal transform process on the transform coefficient data inputfrom the dequantization section 21 to thereby restore the predictionerror data. Then, the inverse orthogonal transform section 22 outputsthe restored prediction error data to the addition section 23.

The addition section 23 adds the restored prediction error data inputfrom the inverse orthogonal transform section 22 and the predicted imagedata input from the mode selecting section 50 to thereby generatedecoded image data. Then, the addition section 23 outputs the decodedimage data thus generated to the deblocking filter 24 and the framememory 25.

The deblocking filter 24 applies filtering to reduce blocking artifactsproduced at the time of image encoding. The deblocking filter 24 removesblocking artifacts by filtering the decoded image data input from theaddition section 23, and outputs the decoded image data thus filtered tothe frame memory 25.

The frame memory 25 uses a storage medium to store the decoded imagedata input from the addition section 23 and the decoded image data afterfiltering input from the deblocking filter 24.

The selector 26 reads, from the frame memory 25, unfiltered decodedimage data to be used for intra prediction, and supplies the decodedimage data thus read to the intra prediction section 30 as referenceimage data. Also, the selector 26 reads, from the frame memory 25, thefiltered decoded image data to be used for inter prediction, andsupplies the decoded image data thus read to the motion estimationsection 40 as reference image data.

The intra prediction section 30 performs an intra prediction process ineach intra prediction mode, on the basis of the image data to be encodedthat is input from the syntax processing section 13, and the decodedimage data supplied via the selector 26. For example, the intraprediction section 30 evaluates the prediction result of each intraprediction mode using a predetermined cost function. Then, the intraprediction section 30 selects the intra prediction mode yielding thesmallest cost function value, that is, the intra prediction modeyielding the highest compression ratio, as the optimal intra predictionmode. Furthermore, the intra prediction section 30 outputs informationabout intra prediction, such as prediction mode information indicatingthe optimal intra prediction mode, the predicted image data and the costfunction value, to the mode selecting section 50.

The motion estimation section 40 performs an inter prediction process(prediction process between frames) on the basis of image data to beencoded that is input from the syntax processing section 13, and decodedimage data supplied via the selector 26. For example, the motionestimation section 40 evaluates the prediction result of each predictionmode using a predetermined cost function. Then, the motion estimationsection 40 selects the prediction mode yielding the smallest costfunction value, that is, the prediction mode yielding the highestcompression ratio, as the optimal prediction mode. The motion estimationsection 40 generates predicted image data according to the optimalprediction mode. The motion estimation section 40 outputs informationabout inter prediction, such as prediction mode information indicatingthe optimal intra prediction mode thus selected, the predicted imagedata and the cost function value, to the mode selecting section 50.

The mode selecting section 50 compares the cost function value relatedto intra prediction input from the intra prediction section 30 to thecost function value related to inter prediction input from the motionestimation section 40. Then, the mode selecting section 50 selects theprediction method with the smaller cost function value between intraprediction and inter prediction. In the case of selecting intraprediction, the mode selecting section 50 outputs the information aboutintra prediction to the lossless encoding section 17, and also outputsthe predicted image data to the subtraction section 14 and the additionsection 23. Also, in the case of selecting inter prediction, the modeselecting section 50 outputs the information about inter predictiondescribed above to the lossless encoding section 17, and also outputsthe predicted image data to the subtraction section 14 and the additionsection 23.

1-2. Exemplary Configuration of Syntax Processing Section

FIG. 2 is a block diagram illustrating an example of a detailedconfiguration of the syntax processing section 13 of the image encodingdevice 10 illustrated in FIG. 1. Referring to FIG. 2, the syntaxprocessing section 13 includes a settings section 110, a parametergenerating section 120, and an inserting section 130.

(1) Settings Section

The settings section 110 retains various settings used for the encodingprocess by the image encoding device 10. For example, the settingssection 110 retains a profile for each sequence in the image data, theencoding mode for each picture, data regarding the GOP structure, andthe like. Also, in the present embodiment, the settings section 110retains settings regarding quantization matrices used by thequantization section 16 (and the inverse quantization section 21). Thequestion of which quantization matrix should be used by the quantizationsection 16 may be predetermined for each slice, typically on the basisof offline image analysis.

For example, in exemplary applications such as digital video cameras,compression artifacts do not exist in the input images, and thus aquantization matrix with a reduced quantization step may be used, evenin the high range. The quantization matrix varies in picture units orframe units. In the case of input images with low complexity, using aflat quantization matrix with a smaller quantization step enablesimprovement in the image quality subjectively perceived by the user. Onthe other hand, in the case of input images with high complexity, it isdesirable to use a larger quantization step in order to inhibitincreases in the rate. In this case, using a flat quantization matrixhas the risk of artifacts in the low range signal being recognized asblock noise. For this reason, it is beneficial to reduce noise by usinga quantization matrix in which the quantization step increasesproceeding from the low range to the high range.

In exemplary applications such as recorders that recompress broadcastcontent encoded in MPEG-2, MPEG-2 compression artifacts such as mosquitonoise exist in the input images themselves. Mosquito noise is noiseproduced as a result of quantizing a high range signal with a largerquantization step, and the frequency components of the noise becomeextremely high frequencies themselves. When recompressing such inputimages, it is desirable to use a quantization matrix having a largequantization step in the high range. Also, in interlaced signals, thecorrelation of signal in the horizontal direction is higher than thecorrelation of the signal in the vertical direction compared toprogressive signals, due to the effects of the interlaced scans. Forthis reason, it is also beneficial to use different quantizationmatrices according to whether the image signal is a progressive signalor an interlaced signal. In either case, the optimal quantization matrixmay vary in picture units or frame units, depending on the imagecontent.

(2) Parameter Generating Section

The parameter generating section 120 generates parameters definingsettings for the encoding process which are retained by the settingssection 110, and outputs the generated parameters to the insertingsection 130.

For example, in the present embodiment, the parameter generating section120 generates quantization matrix parameters defining quantizationmatrices to be used by the quantization section 16. The group ofquantization matrix parameters generated by the parameter generatingsection 120 are included in the quantization matrix parameter set(QMPS). Each QMPS is assigned a QMPS ID, which is an identifier fordiscriminating the individual QMPSs from each other. Typically, multipleclasses of quantization matrices are defined inside one QMPS. Theclasses of quantization matrices are distinguished from each other bythe matrix size along with the corresponding prediction method, and thesignal components. For example, a maximum of six classes of quantizationmatrices (the Y/Cb/Cr components in intra prediction/inter prediction)may be defined for each of the sizes 4×4, 8×8, 16×16, and 32×32 insideone QMPS.

More specifically, the parameter generating section 120 may convert eachquantization matrix into a linear array using a zigzag scan, and encodethe value of each element in the linear array in differential pulse-codemodulation (DPCM) format, similarly to the quantization matrix encodingprocess in H.264/AVC. In this case, the linear arrays of DPCMdifferential data become the quantization matrix parameters. In thisspecification, such a mode for generating quantization matrix parametersis designated the full scan mode.

In addition, the parameter generating section 120 may also generatequantization matrix parameters in a mode that differs from the full scanmode, in order to reduce the amount of codes of the quantization matrixparameters. For example, instead of the full scan mode, the parametergenerating section 120 may also generate quantization matrix parametersin a copy mode or an axis designation mode, described next.

Copy mode is a mode that may be selected in the case where thequantization matrix used for a given slice resembles or equals analready-defined quantization matrix. In the case of copy mode, theparameter generating section 120 includes the QMPS ID of the QMPS inwhich the copy source quantization matrix is defined, as well as thesize and type of the copy source quantization matrix, into the QMPS asquantization matrix parameters. Note that in this specification, thecombination of the prediction method and signal components correspondingto a given quantization matrix is designated the type of thatquantization matrix. In the case where differences exist between thequantization matrix to be defined and the copy source quantizationmatrix, the parameter generating section 120 may additionally includeresidual data for generating a residual matrix expressing the residualerror of each component in the QMPS.

Processing for axis designation mode is further divided according to twospecifying methods: a differential method and an interpolation method.With the differential method, the parameter generating section 120specifies only the values of the elements in the quantization matrixcorresponding to the vertical axis which is the leftmost column, thehorizontal axis which is the uppermost row, and diagonal axis along thediagonal of the transform unit. With the interpolation method, theparameter generating section 120 specifies only the values of theelements in the quantization matrix corresponding to the four corners atthe upper-left (the DC component), the upper-right, the lower-left, andthe lower-right of the transform unit.

The values of the remaining elements may be interpolated with anarbitrary technique such as linear interpolation, cubic interpolation,or Lagrange interpolation. Likewise in axis designation mode, in thecase where differences exist between the quantization matrix to bedefined and the interpolated quantization matrix, the parametergenerating section 120 may additionally include residual data forgenerating a residual matrix expressing the residual error of eachcomponent in the QMPS.

(3) Inserting Section

The inserting section 130 inserts header information, such as SPSs,PPSs, QMPSs, and slice headers that respectively include the parametergroups generated by the parameter generating section 120, into thestream of image data input from the reordering buffer 12. As discussedearlier, the QMPS is a non-VCL NAL unit that differs from the SPS andthe PPS. The QMPS includes quantization matrix parameters generated bythe parameter generating section 120. The inserting section 130 thenoutputs the stream of image data with inserted header information to thesubtraction section 14, the intra prediction section 30, and the motionestimation section 40.

1-3. Exemplary Parameter Structure

(1) Quantization Matrix Parameter Set

FIG. 3 is an explanatory diagram illustrating exemplary parametersincluded in each QMPS in the present embodiment. Referring to FIG. 3,each QMPS includes a “QMPS ID”, a “generation mode present flag”, a“generation mode”, and different quantization matrix parameters for eachmode.

The “QMPS ID” is an identifier for discriminating the individual QMPSsfrom each other. The QMPS ID may be an integer in the range from 0 to31, or the like. The specification of an unused QMPS ID means that a newQMPS is to be defined. The re-specification of a QMPS ID already beingused in a sequence means that an already-defined QMPS is to be updated.

The “generation mode present flag” is a flag indicating whether or not a“generation mode”, which is a classification representing a mode of thequantization matrix generation process, is present inside that QMPS. Inthe case where the generation mode present flag indicates “0: notpresent”, quantization matrices are defined in full scan mode insidethat QMPS. Meanwhile, in the case where the generation mode present flagindicates “1: present”, a “generation mode” is present inside that QMPS.

The “generation mode” is a classification that may take any of thevalues “0: copy”, “1: axis designation”, or “2: full scan”, for example.In the syntax pseudo-code described later, the generation mode isrepresented by a variable called “pred_mode”.

In the case of copy mode (that is, pred_mode=0), the QMPS may include a“source ID”, a “copy source size”, a “copy source type”, a “residualflag”, and “residual data” as quantization matrix parameters. The“source ID” is a QMPS ID specifying a QMPS in which a copy sourcequantization matrix is defined. The “copy source size” is the size ofthe copy source quantization matrix. The “copy source type” is the typeof the copy source quantization matrix (intra-Y, intra-Cb, . . . ,intra-Cr). The “residual flag” is a flag indicating whether or notresidual error is present. The “residual data” is data for generating aresidual matrix expressing the residual error in the case where residualerror is present. The residual data may be omitted in the case where theresidual flag indicates “0: not present”.

Note the case where the source ID of a given QMPS is equal to QMPS ID ofthat QMPS itself may be interpreted as specifying a default quantizationmatrix like that illustrated by example in FIG. 38. Doing so may reducethe amount of codes of the QMPS, since an independent flag forspecifying the default quantization matrix no longer needs to beincluded in the QMPS.

In the case of axis designation mode (that is, pred_mode=1), the QMPSmay include a “designation method flag”, in addition to either“reference axis data” or “corner data”, a “residual flag”, and “residualdata” as quantization matrix parameters. The designation method flag isa flag that indicates how to designate the values of elements alongreference axes that serve as a reference for the generation of aquantization matrix, and may take a value of either “0: differential” or“1: interpolation”, for example. In the case where the designationmethod is “0: differential”, the values of elements corresponding to thereference axes of the quantization matrix, these being the verticalaxis, the horizontal axis, and the diagonal axis, are designated by thereference axis data. In the case where the designation method is “1:interpolation”, the values of elements corresponding to the four cornersat the upper-left, the upper-right, the lower-left, and the lower-rightof the quantization matrix are designated by the corner data. The valuesof elements on the three reference axes may be generated byinterpolation from the values of these four corners. The residual flagand the residual data are similar to the case of copy mode.

In the case of full scan mode (that is, pred_mode=2), the QMPS mayinclude linear arrays of DPCM differential data as quantization matrixparameters.

Note that each QMPS may include generation modes that differ for eachclass of quantization matrix and quantization matrix parameterscorresponding to each mode. In other words, as an example, a given classof quantization matrix may be defined in full scan mode, another classof quantization matrix may be defined in axis designation mode, and theremaining quantization matrices may be defined in copy mode inside asingle QMPS.

(2) Slice Header

FIG. 4 is an explanatory diagram partially illustrating exemplaryparameters included in each slice header in the present embodiment.Referring to FIG. 4, each slice header may include a “slice type”, a“PPS ID”, a “QMPS ID present flag”, and a “QMPS ID”. The “slice type” isa classification indicating the encoding type for that slice, and takesa value corresponding to a P slice, B slice, or I slice, or the like.The “PPS ID” is an ID for the picture parameter set (PPS) referenced forthat slice. The “QMPS ID present flag” is a flag indicating whether ornot a QMPS ID is present in that slice header. The “QMPS ID” is a QMPSID for the quantization matrix parameter set (QMPS) referenced for thatslice.

<2. Process Flow During Encoding According to Embodiment>

(1) Parameter Set Insertion Process

FIG. 5 is a flowchart illustrating an exemplary flow of a parameter setinsertion process by the inserting section 130 of the syntax processingsection 13 according to the present embodiment.

Referring to FIG. 5, the inserting section 130 first successivelyacquires NAL units inside the stream of image data input from thereordering buffer 12, and recognizes a single picture (step S100). Next,the inserting section 130 determines whether or not the recognizedpicture is the leading picture of a sequence (step S102). At this point,the inserting section 130 inserts an SPS into the stream in the casewhere the recognized picture is the leading picture of a sequence (stepS104). Next, the inserting section 130 additionally determines whetheror not there is a change in the PPS for the recognized picture (stepS106). At this point, the inserting section 130 inserts a PPS into thestream in the case where there is a change in the PPS, or in the casewhere the recognized picture is the leading picture of a sequence (stepS108). Next, the inserting section 130 additionally determines whetheror not there is a change in the QMPS (step S110). At this point, theinserting section 130 inserts a QMPS into the stream in the case wherethere is a change in the QMPS, or in the case where the recognizedpicture is the leading picture of a sequence (step S112). After that,the inserting section 130 ends the process in the case of detecting theend of the stream. On the other hand, the inserting section 130 repeatsthe above process for the next picture in the case where the stream hasnot ended (step S114).

Note that although the flowchart only illustrates the insertion of theSPS, the PPS, and QMPS for the sake of simplicity, the inserting section130 may also insert other header information, such as supplementalenhancement information (SEI) and slice headers, into the stream.

(2) Description of Stream Structure

FIG. 6 is an explanatory diagram for explaining differences in thestream structure between a technique according to the present embodimentand an existing technique.

The left side of FIG. 6 illustrates a stream ST1 as an example generatedin accordance with an existing technique. Since the start of the streamST1 is the start of a sequence, a first SPS(1) and a first PPS(1) areinserted at the start of the stream ST1. One or more quantizationmatrices may be defined in the SPS(1) and the PPS(1). Next, assume thatit becomes necessary to update the quantization matrices after severalsubsequent slice headers and slice data. Thus, a second PPS(2) isinserted into the stream ST1. The PPS(2) also includes parameters otherthan quantization matrix parameters. Next, assume that it becomesnecessary to update the PPS after several subsequent slice headers andslice data. Thus, a third PPS(3) is inserted into the stream ST1. ThePPS(3) also includes quantization matrix parameters. The quantizationprocess (and inverse quantization process) for subsequent slices isconducted using the quantization matrices defined in the PPS specifiedby the PPS ID in the slice header.

The right side of FIG. 6 illustrates a stream ST2 as an examplegenerated in accordance with the above technique according to thepresent embodiment. Since the start of the stream ST2 is the start of asequence, a first SPS(1), a first PPS(1), and a first QMPS(1) areinserted at the start of the stream ST1. In the stream ST2, one or morequantization matrices may be defined in the QMPS(1). The sum of thelengths of the PPS(1) and the QMPS(1) in the stream ST2 areapproximately equal to the length of the PPS(1) in the stream ST1. Next,if it becomes necessary to update the quantization matrices afterseveral subsequent slice headers and slice data, a second QMPS(2) isinserted into the stream ST2. Since the QMPS(2) does not containparameters other than quantization matrix parameters, the length of theQMPS(2) is shorter than the length of the PPS(2) in the stream ST2.Next, if it becomes necessary to update the PPS after several subsequentslice headers and slice data, a second PPS(2) is inserted into thestream ST2. Since the PPS(2) in the stream ST2 does not containquantization matrix parameters, the length of the PPS(2) in the streamST2 is shorter than the length of the PPS(3) in the stream ST1. Thequantization process (and inverse quantization process) for subsequentslices is conducted using the quantization matrices defined in the QMPSspecified by the QMPS ID in the slice header.

A comparison of the streams ST1 and ST2 in FIG. 6 reveals that theamount of codes of the stream overall may be reduced with the techniquedescribed in the present embodiment. Particularly, the reduction of theamount of codes with the above technique becomes even more effective inthe case of quantization matrices with larger sizes, or in the casewhere a larger number of quantization matrices are defined for eachpicture.

3. EXEMPLARY CONFIGURATION OF IMAGE DECODING DEVICE ACCORDING TOEMBODIMENT 3-1. Exemplary Overall Configuration

This section describes an exemplary configuration of an image decodingdevice according to an embodiment.

3-1. Exemplary Overall Configuration

FIG. 7 is a block diagram illustrating an exemplary configuration of animage decoding device 60 according to an embodiment. Referring to FIG.7, the image decoding device 60 is equipped with a syntax processingsection 61, a lossless decoding section 62, an inverse quantizationsection 63, an inverse orthogonal transform section 64, an additionsection 65, a deblocking filter 66, a reordering buffer 67, adigital-to-analog (D/A) conversion section 68, frame memory 69,selectors 70 and 71, an intra prediction section 80, and a motioncompensation section 90.

The syntax processing section 61 acquires header information such asSPSs, PPSs, QMPSs, and slice headers from an encoded stream input via atransmission channel, and recognizes various settings for a decodingprocess by the image decoding device 60 on the basis of the acquiredheader information. For example, in the present embodiment, the syntaxprocessing section 61 sets a quantization matrix to be used during aninverse quantization process by the inverse quantization section 63 onthe basis of quantization matrix parameters included in a QMPS. Adetailed configuration of the syntax processing section 61 will befurther described later.

The lossless decoding section 62 decodes the encoded stream input fromthe syntax processing section 61 according to the coding method used atthe time of encoding. The lossless decoding section 62 then outputs thedecoded quantization data to the inverse quantization section 62. Inaddition, the lossless decoding section 62 outputs information aboutintra prediction included in the header information to the intraprediction section 80, and outputs information about inter prediction tothe motion compensation section 90.

The inverse quantization section 63 uses a quantization matrix set bythe syntax processing section 61 to inversely quantize the quantizationdata decoded by the lossless decoding section 62 (that is, quantizedtransform coefficient data). The question of which quantization matrixshould be used for each block in a given slice may be determinedaccording to the QMPS ID specified in the slice header, the size of eachblock (Transform Unit), the prediction method for each block, and thesignal components.

The inverse orthogonal transform section 64 generates prediction errordata by performing an inverse orthogonal transform on transformcoefficient data input from the dequantization section 63 according tothe orthogonal transform method used at the time of encoding. Then, theinverse orthogonal transform section 64 outputs the generated predictionerror data to the addition section 65.

The addition section 65 adds the prediction error data input from theinverse orthogonal transform section 64 to predicted image data inputfrom the selector 71 to thereby generate decoded image data. Then, theaddition section 65 outputs the decoded image data thus generated to thedeblocking filter 66 and the frame memory 69.

The deblocking filter 66 removes blocking artifacts by filtering thedecoded image data input from the addition section 65, and outputs thedecoded image data thus filtered to the reordering buffer 67 and theframe memory 69.

The reordering buffer 67 generates a chronological sequence of imagedata by reordering images input from the deblocking filter 66. Then, thereordering buffer 67 outputs the generated image data to the D/Aconversion section 68.

The D/A conversion section 68 converts the image data in a digitalformat input from the reordering buffer 67 into an image signal in ananalog format. Then, the D/A conversion section 68 causes an image to bedisplayed by outputting the analog image signal to a display (notillustrated) connected to the image decoding device 60, for example.

The frame memory 69 uses a storage medium to store the unfiltereddecoded image data input from the addition section 65 and the filtereddecoded image data input from the deblocking filter 66.

The selector 70 switches the output destination of the image data fromthe frame memory 69 between the intra prediction section 80 and themotion compensation section 90 for each block in the image according tomode information acquired by the lossless decoding section 62. Forexample, in the case where an intra prediction mode is specified, theselector 70 outputs the unfiltered decoded image data that is suppliedfrom the frame memory 69 to the intra prediction section 80 as referenceimage data. Also, in the case where an inter prediction mode isspecified, the selector 70 outputs the filtered decoded image data thatis supplied from the frame memory 69 to the motion compensation section90 as reference image data.

The selector 71 switches the output source of predicted image data to besupplied to the addition section 65 between the intra prediction section80 and the motion compensation section 90 for each block in the imageaccording to the mode information acquired by the lossless decodingsection 62. For example, in the case where an intra prediction mode isspecified, the selector 71 supplies the addition section 65 with thepredicted image data output from the intra prediction section 80. In thecase where an inter prediction mode is specified, the selector 71supplies the addition section 65 with the predicted image data outputfrom the motion compensation section 90.

The intra prediction section 80 performs in-picture prediction of pixelvalues on the basis of the information about intra prediction input fromthe lossless decoding section 62 and the reference image data from theframe memory 69, and generates predicted image data. Then, the intraprediction section 80 outputs the predicted image data thus generated tothe selector 71.

The motion compensation section 90 performs a motion compensationprocess on the basis of the information about inter prediction inputfrom the lossless decoding section 62 and the reference image data fromthe frame memory 69, and generates predicted image data. Then, themotion compensation section 90 outputs the predicted image data thusgenerated to the selector 71.

3-2. Exemplary Configuration of Syntax Processing Section

FIG. 8 is a block diagram illustrating an example of a detailedconfiguration of the syntax processing section 61 of the image decodingdevice 60 illustrated in FIG. 7. Referring to FIG. 8, the syntaxprocessing section 61 includes a parameter acquiring section 160 and asetting section 170.

(1) Parameter Acquiring Section

The parameter acquiring section 160 recognizes header information suchas SPSs, PPSs, QMPSs, and slice headers from the stream of image data,and acquires parameters included in the header information. For example,in the present embodiment, the parameter acquiring section 160 acquiresquantization matrix parameters defining a quantization matrix from aQMPS. As discussed earlier, the QMPS is a non-VCL NAL unit that differsfrom the SPS and the PPS. The parameter acquiring section 160 thenoutputs the acquires parameters to the setting section 170. Theparameter acquiring section 160 also outputs the stream of image data tothe lossless decoding section 62.

(2) Setting Section

The setting section 170 applies settings for the processing in eachsection illustrated in FIG. 7 on the basis of the parameters acquired bythe parameter acquiring section 160. For example, the setting section170 recognizes the range of Coding Unit sizes from the pair of LCU andSCU values, while also setting the Coding Unit size according to thevalue of split_flag. The decoding of image data is conducted by takingthe Coding Units set at this point as the units of processing. Inaddition, the setting section 170 additionally sets the Transform Unitsize. The inverse quantization by the inverse quantization section 63and the inverse orthogonal transform by the inverse orthogonal transformsection 64 discussed above are conducted by taking the Transform Unitsset at this point as the units of processing.

Also, in the present embodiment, the setting section 170 setsquantization matrices on the basis of quantization matrix parametersacquired from a QMPS by the parameter acquiring section 160. Morespecifically, on the basis of quantization parameters included in aQMPS, the setting section 170 generates multiple quantization matricesthat differ from each other in size and type in full scan mode, copymode, and axis designation mode, respectively. The generation ofquantization matrices may be conducted each time a QMPS is detected inthe stream of image data.

For example, in full scan mode, the setting section 170 decodes a lineararray of differential data included in the quantization matrixparameters in DPCM format. The setting section 170 then converts thedecoded linear array to a two-dimensional quantization matrix accordingto the scan pattern of a zigzag scan.

Also, in copy mode, the setting section 170 copies a (previouslygenerated) quantization matrix specified by a source ID, a copy sourcesize, and a copy source type included in the quantization matrixparameters. At this point, in the case where the size of the newquantization matrix is smaller than the size of the copy sourcequantization matrix, the setting section 170 generates the newquantization matrix by decimating elements in the copied quantizationmatrix. In addition, in the case where the size of the new quantizationmatrix is larger than the size of the copy source quantization matrix,the setting section 170 generates the new quantization matrix byinterpolating elements in the copied quantization matrix. Then, in thecase where residual components are present, the setting section 170 addsthe residual components to the new quantization matrix.

Also, in the case where the source ID included in the quantizationmatrix parameters inside a given QMPS is equal to the QMPS ID of thatQMPS, the setting section 170 treats the new quantization matrix as thedefault quantization matrix.

Also, in axis designation mode, the setting section 170 recognizes thedesignation method flag included in the quantization matrix parameters.Then, in the case of the differential method, the setting section 170generates the values of the elements of the quantization matrix thatcorrespond to the vertical axis, the horizontal axis, and the diagonalaxis on the basis of reference axis data included in the quantizationmatrix parameters, and generates the values of the remaining elements byinterpolation. In addition, in the case of the interpolation method, thesetting section 170 generates the values of the elements of thequantization matrix that correspond to the four corners on the basis ofcorner data included in the quantization matrix parameters, and aftergenerating the values of elements along the reference axes byinterpolation, additionally generates the values of the remainingelements by interpolation. Then, in the case where residual componentsare present, the setting section 170 adds the residual components to thenew quantization matrix.

After that, when a QMPS ID is specified in a slice header, the settingsection 170 sets the quantization matrix generated for the QMPSidentified by the specified QMPS ID as the quantization matrix to beused by the inverse quantization section 63.

4. PROCESS FLOW DURING DECODING ACCORDING TO EMBODIMENT 4-1. GeneratingQuantization Matrices

(1) Quantization Matrix Generation Process

FIG. 9 is a flowchart illustrating an exemplary flow of a quantizationmatrix generation process by the syntax processing section 61 accordingto the present embodiment. The quantization matrix generation process inFIG. 9 is a process that may be conducted each time a QMPS is detectedin the stream of image data.

Referring to FIG. 9, the parameter acquiring section 160 first acquiresa QMPS ID from a QMPS (step S200). If the QMPS ID acquired at this pointis an unused ID in the stream, the setting section 170 generates a newquantization matrix to be associated with that QMPS ID according to theprocess described below. On the other hand, if the QMPS ID is an ID thatis already in use, the setting section 170 updates the quantizationmatrix stored in association with that QMPS ID to a matrix generatedaccording to the process described below. Next, the parameter acquiringsection 160 acquires a generation mode present flag from the QMPS (stepS202).

The subsequent processing from step S206 to step S240 is repeated forevery class of quantization matrix (step S204). Note that the class of aquantization matrix corresponds to the combination of the size and thetype (that is, the prediction method and signal components) of thequantization matrix.

In step S206, the setting section 170 determines, according to thegeneration mode present flag, whether or not (a classification of) ageneration mode is present in the QMPS (step S206). In the case where ageneration mode is not present at this point, the setting section 170generates a quantization matrix with the full scan method, similarly tothe quantization matrix decoding process in H.264/AVC (step S208). Onthe other hand, in the case where a generation mode is present, theparameter acquiring section 160 acquires the generation mode from theQMPS (step S210). The setting section 170 then conducts differentprocessing depending on the generation mode (steps S212, S214).

For example, in the case where copy mode is indicated, the settingsection 170 conducts processing in the copy mode illustrated by examplein FIG. 10 (step S220). Also, in the case where axis designation mode isindicated, the setting section 170 conducts processing in the axisdesignation mode illustrated by example in FIG. 11 (step S240). Also, inthe case where full scan mode is indicated, the setting section 170generates a quantization matrix with the full scan method, similarly tothe quantization matrix decoding process in H.264/AVC (step S208).

After that, when quantization matrices for all classes of quantizationmatrices are generated, the quantization matrix generation processillustrated in FIG. 9 ends.

(2) Process in Copy Mode

FIG. 10 is a flowchart illustrating an example of a detailed flow of aprocess in copy mode in step S220 of FIG. 9.

Referring to FIG. 10, first, the parameter acquiring section 160acquires a source ID from a QMPS (step S221). Next, the setting section170 determines whether or not the QMPS ID acquired in step S200 of FIG.9 (the QMPS ID of the current QMPS) and the source ID are equal (stepS222). At this point, in the case where the QMPS ID of the current QMPSand the source ID are equal, the setting section 170 generates a newquantization matrix as the default quantization matrix (step S223). Onthe other hand, in the case where the QMPS ID of the current QMPS andthe source ID are not equal, the process proceeds to step S224.

In step S224, the parameter acquiring section 160 acquires the copysource size and the copy source type from the QMPS (step S224). Next,the setting section 170 copies the quantization matrix specified by thesource ID, the copy source size, and the copy source type (step S225).Next, the setting section 170 compares the copy source size to the sizeof the quantization matrix to be generated (step S226). At this point,in the case where the copy source size and the size of the quantizationmatrix to be generated are not equal, the setting section 170 generatesa new quantization matrix by interpolating or decimating elements in thecopied quantization matrix, depending on the size difference (stepS227).

Additionally, the parameter acquiring section 160 acquires the residualflag from the QMPS (step S228). Next, the setting section 170 determineswhether or not residual data is present, according to the value of theresidual flag (step S229). At this point, in the case where residualdata is present, the setting section 170 adds the residual error to thenew quantization matrix generated in step S223 or steps S225 to S227(step S230).

(3) Process in Axis Designation Mode

FIG. 11 is a flowchart illustrating an example of a detailed flow of aprocess in axis designation mode in step S240 of FIG. 9.

Referring to FIG. 11, first, the parameter acquiring section 160acquires the designation method flag from a QMPS (step S241). Next, thesetting section 170 determines the designation method according to thevalue of the designation method flag (step S242). At this point, theprocess proceeds to step S243 in the case where the differential methodis designated. On the other hand, the process proceeds to step S246 inthe case where the interpolation method is designated.

In the case of the differential method, the setting section 170generates the values of the elements of the quantization matrix thatcorrespond to the vertical axis, the horizontal axis, and the diagonalaxis, on the basis of reference axis data included in the quantizationmatrix parameters (steps S243, S244, and S245). Meanwhile, in the caseof the interpolation method, the setting section 170 generates thevalues of the elements of the quantization matrix that correspond to thefour corners, on the basis of corner data included in the quantizationmatrix parameters (step S246). Next, the setting section 170 generatesby interpolation the values of the elements along the reference axes(vertical axis, horizontal axis, and diagonal axis) that join the fourcorners (step S247). After that, the setting section 170 interpolatesthe values of the remaining elements on the basis of the values of theelements along the reference axes (step S248).

Additionally, the parameter acquiring section 160 acquires the residualflag from the QMPS (step S249). Next, the setting section 170 determineswhether or not residual data is present, according to the value of theresidual flag (step S250). At this point, in the case where residualdata is present, the setting section 170 adds the residual error to thenew quantization matrix generated in step S248 (step S251).

4-2. Setting Quantization Matrix to Slice

FIG. 12 is a flowchart illustrating an exemplary flow of a process forsetting a quantization matrix to a slice by the syntax processingsection 61 according to the present embodiment. The process in FIG. 12may be conducted each time a slice header is detected in the stream ofimage data.

First, the parameter acquiring section 160 acquires the QMPS ID presentflag from a slice header (step S261). Next, the parameter acquiringsection 160 determines whether or not a QMPS ID is present inside theslice header, according to the value of the QMPS ID present flag (stepS262). At this point, the parameter acquiring section 160 additionallyacquires a QMPS ID from a QMPS in the case where a QMPS ID is present(step S263). The setting section 170 then sets the quantization matrixgenerated for the QMPS identified by the acquired QMPS ID for slicesfollowing that slice header (step S264). On the other hand, in the casewhere a QMPS ID is not present inside the slice header, the settingsection 170 sets a flat quantization matrix for slices following thatslice header (step S265).

5. SYNTAX EXAMPLES 5-1. First Example

FIGS. 13 to 15 illustrate a first example of illustrative pseudo-codeexpressing the syntax of a QMPS according to the present embodiment.Line numbers are given on the left edge of the pseudo-code. Also, anunderlined variable in the pseudo-code means that the parametercorresponding to that variable is specified inside the QMPS.

The function QuantizationMatrixParameterSet( ) on line 1 in FIG. 13 is afunction that expresses the syntax of a single QMPS. On line 2 and line3, the QMPS ID (quantization_matrixparamter_id) and the generation modepresent flag (pred_present_flag) are specified. The subsequent syntaxfrom line 6 to line 56 loops for every size and type of quantizationmatrix. The syntax from line 7 to line 53 in the loop is inserted intothe QMPS in the case where a generation mode is present(pred_present_flag=1).

The syntax from line 9 to line 16 in the case where a generation mode ispresent is the syntax for copy mode. From line 9 to line 11, the sourceID, copy source size, and copy source type are specified. The functionpred_matrix( ) on line 12 means that the quantization matrix specifiedby the source ID, copy source size, and copy source type is to becopied. On line 13, the residual flag is specified. The functionresidual_matrix( ) on line 15 means that residual data is specified inthe QMPS in the case where residual components are present.

The syntax from line 18 to line 50 is the syntax for axis designationmode, and is described in FIG. 14. On line 18, the designation methodflag is specified. In the case where the designation method is thedifferential (DPCM) method, the values of elements along the verticalaxis are specified from line 21 to line 25, while the values of elementsalong the horizontal axis are specified from line 26 to line 34, and thevalues of elements along the diagonal axis are specified from line 35 toline 40. The reference axis data in this case are linear arrays of DPCMdifferential data. Note that the syntax for elements along thehorizontal axis may be omitted in the case where the values of elementsalong the horizontal axis may be copied (line 27, line 28). In the casewhere the designation method is the interpolation method, the values ofthe upper-left (DC component), upper-right, lower-left, and lower-rightelements are respectively specified as corner data from line 42 to line45.

The processing on line 52 is the syntax for the full scan mode. Theprocessing on line 55 is the syntax for the case where a generation modeis not present. In either case, a quantization matrix is specified withthe full scan method by a function qmatrix( ) that represents thequantization matrix syntax in H.264/AVC.

The function residual_matrix( ) on line 1 in FIG. 15 is a function forspecifying residual data used on line 15 in FIG. 13 and line 49 in FIG.14. In the example in FIG. 15, residual data is specified by a DPCMmethod or a run-length method. In the case of the DPCM method, the valueof the difference from the last element (delta_coef) is specified fromline 4 to line 8 for every element in the linear array. In the case ofthe run-length method, the length of element groups in portions in whichthe value is consecutively zero (run) and the value of non-zero elements(data) are repeatedly specified from line 11 to line 18.

5-2. Second Example

FIGS. 16 to 20 illustrate a second example of illustrative pseudo-codeexpressing the syntax of a QMPS according to the present embodiment.

The function QuantizationMatrixParameterSet( ) on line 1 in FIG. 16 is afunction that expresses the syntax of a single QMPS. On line 2, the QMPSID (quantization_matrix_paramter_id) is specified. In addition, thegeneration mode present flag (pred_present flag) is specified on line 6,except in the case where only the default quantization matrix isdesignated.

Furthermore, in the second example, four classes of quantization scales(Qscale0 to Qscale3) are specified from line 7 to line 10 in thefunction QuantizationMatrixParameterSet( ). These quantization scalesare parameters that may be adopted in order to quantize the value ofeach element in a quantization matrix and further decrease the rate.More specifically, four quantization scale setting areas A1 to A4 likethose illustrated in FIG. 21 are defined in an 8×8 quantization matrix,for example. The quantization scale setting area A1 is an area for theelement group corresponding to the low-range signal, including the DCcomponent. The quantization scale setting areas A2 and A3 are areas forthe element groups that correspond to respective signals in themid-range. The quantization scale setting area A4 is an area for theelement group corresponding to the high-range signal. A quantizationscale for quantizing the values of elements in the quantization matrixmay be set for each of these areas. For example, referring to FIG. 22,the first quantization scale (Qscale0) is “1” for the quantization scalesetting area A1. This means that values in the quantization matrix arenot quantized in element groups corresponding to the low-range signal.Meanwhile, the second quantization scale (Qscale1) is “2” for thequantization scale setting area A2. The third quantization scale(Qscale2) is “3” for the quantization scale setting area A3. The fourthquantization scale (Qscale3) is “4” for the quantization scale settingarea A4. As the quantization scale becomes larger, the error produced byquantization increases. Typically, however, some degree of error istolerable for the high-range signal. Consequently, in cases where it isdesirable to achieve a high encoding efficiency, the amount of codesrequired to define a quantization matrix can be effectively reduced bysetting such quantization scales for quantizing the quantization matrix,without greatly degrading image quality. In cases where a quantizationmatrix is quantized, the value of each element in the residual data orthe differential data illustrated by example in FIG. 3 may besubstantially quantized or inversely quantized by the quantization stepsset in the quantization scale setting areas to which each elementbelongs.

Note that the layout of quantization scale setting areas illustrated inFIG. 21 is merely one example. For example, different numbers ofquantization scale setting areas may also be defined for each size ofquantization matrix (for example, more quantization scale setting areasmay be defined for larger sizes). Also, the positions of the boundariesof the quantization scale setting areas are not limited to the examplein FIG. 21. Ordinarily, the scan pattern when linearizing a quantizationmatrix is a zigzag scan. For this reason, it is preferable to usediagonal area boundaries from the upper-right to the lower-left, asillustrated in FIG. 21. However, area boundaries following along thevertical direction or the horizontal direction may also be used,depending on factors such as inter-element correlation in thequantization matrix and the scan pattern in use. Furthermore, the layoutof quantization scale setting areas (the number of areas, the positionsof boundaries, and the like) may also be adaptively selected from theperspective of encoding efficiency. For example, a smaller number ofquantization scale setting areas may also be selected in the case wherea nearly-flat quantization matrix is defined.

In FIG. 16, the subsequent syntax from line 13 to line 76 loops forevery size and type of quantization matrix. The syntax from line 14 toline 66 (see FIG. 18) in the loop is inserted into the QMPS in the casewhere a generation mode is present (pred_present_flag=1).

The syntax from line 16 to line 22 in the case where a generation modeis present is the syntax for copy mode. From line 16 to line 18, thesource ID, copy source size, and copy source type are specified. On line19, the residual flag is specified. The function residual_matrix( ) online 21 means that residual data is specified in the QMPS in the casewhere residual components are present. The residual data at this pointmay be quantized according to the values of the four classes ofquantization scales (Qscale0 to Qscale3) discussed above. The syntaxfrom line 23 to line 56 is the syntax for axis designation mode, and isdescribed in FIG. 17. The residual data in axis designation mode maylikewise be quantized according to the values of the four classes ofquantization scales (Qscale0 to Qscale3) discussed above (line 55).

The syntax from line 57 to line 66 in FIG. 18 is the syntax for fullscan mode. Also, the syntax from line 68 to line 75 is the syntax forthe case in which a generation mode is not present. In either case, aquantization matrix is specified with the full scan method by thefunction qmatrix( ). However, in the second example, VLC tables forentropy encoding differential data (delta_coef) in the DPCM method orrun values (run) and non-zero element values (data) in the run-lengthmethod are adaptively switched in order to further raise the encodingefficiency. The vlc_table_data on line 61 and line 71 specify the tablenumber of a VLC table selected for the differential data (delta_coef) inthe DPCM method or the non-zero element values (data) in the run-lengthmethod. The vlc_table_run on line 63 and line 73 specify the tablenumber of a VLC table selected for the run values (run) in therun-length method.

The function qmatrix( ) on line 1 of FIG. 19 is the syntax forspecifying a quantization matrix with the full scan method. Line 3 toline 8 in FIG. 19 indicate the syntax for the DPCM method, and thedifferential data (delta_coef) on line 5 is encoded using the VLC tablespecified by vlc_table_data discussed above. Also, line 10 to line 21indicate the syntax for the run-length method, and the run values (run)on line 12 is encoded using the VLC table specified by vlc_table_rundiscussed above. The non-zero element values (data) on line 13 areencoded using the VLC table specified by vlc_table_data discussed above.

FIG. 23 illustrates code word lists in 11 classes of variable-lengthcoding (VLC) tables selectable in a low complexity entropy coding (LCEC)method. The “x” in each code word in FIG. 23 is a given suffix. Forexample, if a value of “15” is encoded with Exp-Golomb coding, a 9-bitcode word of “000010000” is obtained, but if that value is encoded withVLC4, a 5-bit code word of “11111” is obtained. In this way, theencoding efficiency can be raised by selecting a VLC table havingsuffixes with larger numbers of digits in short code words in the caseof encoding many larger values. Among the 11 classes of VLC tables inFIG. 23, VLC4 for example has 4-digit suffixes in 5-bit code words.Also, VLC9 has 4-digit suffixes in 6-bit code words. Consequently, theseVLC tables are suitable in the case of encoding many larger values.

Returning to the quantization matrix syntax, since the differential dataof a linear array of a quantization matrix has many consecutive zeros,the run values in the run-length method produce many larger values,rather than small values such as 0, 1, or 2. On the other hand, thenon-zero element values and the differential data values in therun-length method produce large values only infrequently. Consequently,by switching the VLC table for each differential data designation method(DPCM/run-length) and the class of value (run/non-zero element in thecase of the run-length method) as with the syntax discussed above, theamount of codes required to define a quantization matrix issignificantly reduced.

The function residual_matrix( ) on line 1 of FIG. 20 also implementsadaptive switching of VLC tables. In other words, the vlc_table_data online 7 specifies the table number of a VLC table selected for thedifferential data (delta_coef) in the DPCM method. The differential data(delta_coef) on line 10 is encoded using the VLC table specified on line7. The vlc_table_data on line 15 specifies the table number of a VLCtable selected for the non-zero element values (data) in the run-lengthmethod. The vlc_table_run on line 16 specifies the table number of a VLCtable selected for the run values (run) in the run-length method. Therun values (run) on line 19 are encoded using the VLC table specified byvlc_table_run discussed above. The non-zero element values (data) online 20 are encoded using the VLC table specified by vlc_table_datadiscussed above.

According to the various features of such a QMPS syntax, the amount ofcodes required to define a quantization matrix is effectively reduced,and the encoding efficiency may be improved. However, the syntaxdescribed in this section is merely one example. In other words, part ofthe syntax illustrated by example herein may be reduced or omitted, thesequence of parameters may be changed, or other parameters may be addedto the syntax. Also, several of the features of the syntax described inthis section may also be implemented when defining a quantization matrixin the SPS or PPS rather than the QMPS. In such cases, it is possible toreduce the amount of codes required to define a quantization matrix inthe SPS or PPS.

6. VARIOUS EXEMPLARY CONFIGURATIONS OF PARAMETER SETS

The foregoing describes several specific examples of the syntax for aquantization matrix parameter set (QMPS) that stores quantization matrixparameters. The QMPS substantially may be a dedicated parameter setcontaining quantization matrix parameters only, but may also be a commonparameter set that also contains other parameters relating to encodingtools other than quantization matrices. For example, “AdaptationParameter Set (APS)” (JCTVC-F747r3, Joint Collaborative Team on VideoCoding (JCT-VC) of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11 6thMeeting: Torino, IT, 14-22 Jul. 2011) introduces a new parameter setcalled the adaptation parameter set (APS), and proposes storingparameters relating to the adaptive loop filter (ALF) and the sampleadaptive offset (SAO) in the APS. By additionally including quantizationmatrix parameters in such an APS, it is also possible to substantiallyconfigure the QMPS discussed above. Thus, in this section, severaltechniques for configuring the QMPS by using the APS proposed by“Adaptation Parameter Set (APS)” (JCTVC-F747r3) will be described.

6-1. First Technique

The first technique is a technique that lists all target parametersinside one APS, and references each parameter using an APS ID, anidentifier that uniquely identifies that APS. FIG. 24 illustrates anexample of an encoded stream configured according to the firsttechnique.

Referring to FIG. 24, an SPS 801, a PPS 802, and an APS 803 are insertedat the start of a picture P0 positioned at the start of a sequence. ThePPS 802 is identified by the PPS ID “P0”. The APS 803 is identified bythe APS ID “A0”. The APS 803 includes ALF-related parameters,SAO-related parameters, and quantization matrix parameters (hereinafterdesignated QM-related parameters). A slice header 804 attached to slicedata inside the picture P0 includes a reference PPS ID “P0”, and thismeans that parameters inside the PPS 802 are referenced in order todecode that slice data. Similarly, the slice header 804 includes areference APS ID “A0”, and this means that parameters inside the APS 803are referenced in order to decode that slice data.

An APS 805 is inserted into a picture P1 following the picture P0. TheAPS 805 is identified by the APS ID “A1”. The APS 805 includesALF-related parameters, SAO-related parameters, and QM-relatedparameters. The ALF-related parameters and the SAO-related parametersincluded in the APS 805 have been updated from the APS 803, but theQM-related parameters have not been updated. A slice header 806 attachedto slice data inside the picture P1 includes a reference APS ID “A1”,and this means that parameters inside the APS 805 are referenced inorder to decode that slice data.

An APS 807 is inserted into a picture P2 following the picture P1. TheAPS 807 is identified by the APS ID “A2”. The APS 807 includesALF-related parameters, SAO-related parameters, and QM-relatedparameters. The ALF-related parameters and the QM-related parametersincluded in the APS 807 have been updated from the APS 805, but theSAO-related parameters have not been updated. A slice header 808attached to slice data inside the picture P2 includes a reference APS ID“A2”, and this means that parameters inside the APS 807 are referencedin order to decode that slice data.

FIG. 25 illustrates an example of APS syntax defined according to thefirst technique. On line 2 in FIG. 25, an APS ID for uniquelyidentifying that APS is specified. The APS ID is a parameter setidentifier used instead of the QMPS ID described using FIG. 3. TheALF-related parameters are specified on line 13 to line 17. TheSAO-related parameters are specified on line 18 to line 23. TheQM-related parameters are specified on line 24 to line 28. The“aps_qmatrix_flag” on line 24 is a quantization matrix present flagindicating whether QM-related parameters are set inside that APS. The“qmatrix_param( )” on line 27 is a function specifying quantizationmatrix parameters as illustrated by example in FIGS. 13 to 20. In thecase where the quantization matrix present flag on line 24 indicatesthat QM-related parameters are set inside that APS (aps_qmatrix_flag=1),the function qmatrix_param( ) may be used to set quantization matrixparameters inside that APS.

In the case of implementing the first technique, the parameter acquiringsection 160 illustrated in FIG. 8 determines whether quantization matrixparameters are set inside an APS by referencing the quantization matrixpresent flag included in that APS. The parameter acquiring section 160then acquires the quantization matrix parameters from the APS in thecase where quantization matrix parameters are set inside that APS.

FIG. 26 is an explanatory diagram illustrating an example of sliceheader syntax defined in accordance with the first technique. On line 5in FIG. 26, there is specified a reference PPS ID for referencingparameters included in the PPS from among the parameters to be set forthat slice. On line 8, there is specified a reference APS ID forreferencing parameters included in the APS from among the parameters tobe set for that slice. The reference APS ID is a reference parameterused instead of the (reference) QMPS ID described using FIG. 4.

According to the first technique, by extending the APS proposed by“Adaptation Parameter Set (APS)” (JCTVC-F747r3), it is possible torealize the quantization matrix parameter set discussed earlier atlittle cost. In addition, it becomes possible to use a quantizationmatrix present flag to partially update only the quantization matrixparameters from among the parameters relating to various encoding toolspotentially included in the APS, or alternatively, partially not updateonly the quantization matrix parameters. In other words, since it ispossible to include quantization matrix parameters in the APS only whenupdating the quantization matrix becomes necessary, quantization matrixparameters can be efficiently transmitted inside the APS.

6-2. Exemplary Modification of First Technique

A technique in accordance with the exemplary modification describedbelow may also be implemented in order to further reduce the amount ofcodes of quantization matrix parameters inside the APS.

FIG. 27 illustrates an example of APS syntax defined according to anexemplary modification of the first technique. In the syntax illustratedin FIG. 27, the QM-related parameters are specified on line 24 to line33. The “aps_qmatrix_flag” on line 24 is a quantization matrix presentflag indicating whether QM-related parameters are set inside that APS.The “ref_aps_idpresent_flag” on line 25 is a past reference ID presentflag indicating whether a past reference ID is to be used as theQM-related parameter in that APS. In the case where the past referenceID present flag indicates that a past reference ID is to be used(ref_aps_idpresent_flag=1), a past reference ID “ref_aps_id” is set online 27. The past reference ID is an identifier for referencing the APSID of an APS encoded or decoded before the current APS. In the casewhere a past reference ID is used, quantization matrix parameters arenot set inside the reference source (latter) APS. In this case, thesetting section 170 illustrated in FIG. 8 reuses the quantizationmatrices set on the basis of the quantization matrix parameters in thereference target APS indicated by the past reference ID as quantizationmatrices corresponding to the reference source APS. Note that a pastreference ID referencing the APS ID of a reference source APS (what iscalled self-reference) may be prohibited. Instead, the setting section170 may set the default quantization matrix as the quantization matrixcorresponding to the self-referencing APS. In the case where a pastreference ID is not used (ref_aps_id_present_flag=0), the function“qmatrix_param( )” on line 31 may be used to set quantization matrixparameters inside that APS.

In this way, by using a past reference ID to reuse already encoded ordecoded quantization matrices, repeatedly setting the same quantizationmatrix parameters inside APSs is avoided. Thus, the amount of codes ofquantization matrix parameters inside the APS can be reduced. Note thatalthough FIG. 27 illustrates an example in which the APS ID is used inorder to reference a past APS, the means of referencing a past APS isnot limited to such an example. For example, another parameter such asthe number of APSs between the reference source APS and the referencetarget APS may also be used in order to reference a past APS. Also,instead of using the past reference ID present flag, the referencing ofa past APS and the setting of new quantization matrix parameters may beswitched depending on whether or not the past reference ID indicates agiven value (minus one, for example.)

6-3. Second Technique

The second technique is a technique that stores parameters in differentAPSs (different NAL units) for each class of parameter, and referenceseach parameter using an APS ID that uniquely identifies each APS. FIG.28 illustrates an example of an encoded stream configured according tothe second technique.

Referring to FIG. 28, an SPS 811, a PPS 812, an APS 813 a, an APS 813 b,and an APS 813 c are inserted at the start of a picture P0 positioned atthe start of a sequence. The PPS 812 is identified by the PPS ID “P0”.The APS 813 a is an APS for ALF-related parameters, and is identified bythe APS ID “A00”. The APS 813 b is an APS for SAO-related parameters,and is identified by the APS ID “A10”. The APS 813 c is an APS forQM-related parameters, and is identified by the APS ID “A20”. A sliceheader 814 attached to slice data inside the picture P0 includes areference PPS ID “P0”, and this means that parameters inside the PPS 812are referenced in order to decode that slice data. Similarly, the sliceheader 814 includes a reference APS_ALF ID “A00”, a reference APS_SAO ID“A10”, and a reference APS_QM ID “A20”, and these mean that parametersinside the APSs 813 a, 813 b, and 813 c are referenced in order todecode that slice data.

An APS 815 a and an APS 815 b are inserted into a picture P1 followingthe picture P0. The APS 815 a is an APS for ALF-related parameters, andis identified by the APS ID “A01”. The APS 815 b is an APS forSAO-related parameters, and is identified by the APS ID “A11”. Since theQM-related parameters are not updated from the picture P0, an APS forQM-related parameters is not inserted. A slice header 816 attached toslice data inside the picture P1 includes a reference APS_ALF ID “A01”,a reference APS_SAO ID “A11”, and a reference APS_QM ID “A20”. Thesemean that parameters inside the APSs 815 a, 815 b, and 813 c arereferenced in order to decode that slice data.

An APS 817 a and an APS 817 c are inserted into a picture P2 followingthe picture P1. The APS 817 a is an APS for ALF-related parameters, andis identified by the APS ID “A02”. The APS 817 c is an APS forQM-related parameters, and is identified by the APS ID “A21”. Since theSAO-related parameters are not updated from the picture P1, an APS forSAO-related parameters is not inserted. A slice header 818 attached toslice data inside the picture P2 includes a reference APS_ALF ID “A02”,a reference APS_SAO ID “A11”, and a reference APS_QM ID “A21”. Thesemean that parameters inside the APSs 817 a, 815 b, and 817 c arereferenced in order to decode that slice data.

The APS for QM-related parameters in the second technique (the APSs 813c and 817 c, for example) are substantially equal to the QMPS discussedearlier. The APS ID of the APS for QM-related parameters is used insteadof the QMPS ID described using FIG. 3. According to the secondtechnique, since different APSs are used for each class of parameters,the transmission of redundant parameters is not conducted for parametersthat do not require updating. For this reason, the encoding efficiencymay be optimized. However, with the second technique, as the classes ofparameters to be incorporated into the APS increase, there is anincrease in the classes of NAL unit types (nal_unit_type), an identifierfor identifying classes of APSs. In the standard specification of HEVC,there are a limited number of NAL unit types (nal_unit_type) reservedfor extensions. Consequently, it is beneficial to consider a structurethat avoids expending many NAL unit types for APSs.

6-4. Third Technique

The third technique is a technique that incorporates quantization matrixparameters and other parameters into the APS, and groups theseparameters by individual identifiers defined separately from the APS ID.In this specification, this identifier assigned to each group anddefined separately from the APS ID is called the sub-identifier (SUBID). Each parameter is referenced using the sub-identifier in the sliceheader. FIG. 29 illustrates an example of an encoded stream configuredaccording to the third technique.

Referring to FIG. 29, an SPS 821, a PPS 822, and an APS 823 are insertedat the start of a picture P0 positioned at the start of a sequence. ThePPS 822 is identified by the PPS ID “P0”. The APS 823 includesALF-related parameters, SAO-related parameters, and QM-relatedparameters. The ALF-related parameters belong to one group, and areidentified by a SUB_ALF ID “AA0”, a sub-identifier for ALF. TheSAO-related parameters belong to one group, and are identified by aSUB_SAO ID “AS0”, a sub-identifier for SAO. The QM-related parametersbelong to one group, and are identified by a SUB_QM ID “AQ0”, asub-identifier for QM. A slice header 824 attached to slice data insidethe picture P0 includes a reference SUB_ALF ID “AA0”, a referenceSUB_SAO ID “AS0”, and a reference SUB_QM ID “AQ0”. These mean that theALF-related parameters belonging to the SUB_ALF ID “AA0”, theSAO-related parameters belonging to the SUB_SAO ID “AS0”, and theQM-related parameters belonging to the SUB_QM ID “AQ0” are referenced inorder to decode that slice data.

An APS 825 is inserted into a picture P1 following the picture P0. TheAPS 825 includes ALF-related parameters and SAO-related parameters. TheALF-related parameters are identified by a SUB_ALF ID “AA1”. TheSAO-related parameters are identified by a SUB_SAO ID “AS1”. Since theQM-related parameters are not updated from the picture P0, QM-relatedparameters are not included in the APS 825. A slice header 826 attachedto slice data inside the picture P1 includes a reference SUB_ALF ID“AA1”, a reference SUB_SAO ID “AS1”, and a reference SUB_QM ID “AQ0”.These mean that the ALF-related parameters belonging to the SUB_ALF ID“AA1” and the SAO-related parameters belonging to the SUB_SAO ID “AS1”inside the APS 825, as well as the QM-related parameters belonging tothe SUB_QM ID “AQ0” inside the APS 823, are referenced in order todecode that slice data.

An APS 827 is inserted into a picture P2 following the picture P1. TheAPS 827 includes ALF-related parameters and QM-related parameters. TheALF-related parameters are identified by a SUB_ALF ID “AA2”. TheQM-related parameters are identified by a SUB_QM ID “AQ1”. Since theSAO-related parameters are not updated from the picture P1, SAO-relatedparameters are not included in the APS 827. A slice header 828 attachedto slice data inside the picture P2 includes a reference SUB_ALF ID“AA2”, a reference SUB_SAO ID “AS1”, and a reference SUB_QM ID “AQ1”.These mean that the ALF-related parameters belonging to the SUB_ALF ID“AA2” and the QM-related parameters belonging to the SUB_QM ID “AQ1”inside the APS 827, as well as the SAO-related parameters belonging tothe SUB_SAO ID “AS1” inside the APS 825, are referenced in order todecode that slice data.

FIG. 30 illustrates an example of APS syntax defined according to thethird technique. On line 2 to line 4 of FIG. 30, three group presentflags “aps_adaptive_loop_filter_flag”,“aps_sample_adaptive_offset_flag”, and “aps_qmatrix_flag” are specified.The group present flags indicate whether or not parameters belonging tothe respective groups are included in that APS. Although the APS ID isomitted from the syntax in the example in FIG. 30, an APS ID foridentifying that APS may also be added within the syntax. TheALF-related parameters are specified on line 12 to line 17. The“sub_alf_id” on line 13 is a sub-identifier for ALF. The SAO-relatedparameters are specified on line 18 to line 24. The “sub_sao_id” on line19 is a sub-identifier for SAO. The QM-related parameters are specifiedon line 25 to line 30. The “sub_qmatrix_id” on line 26 is asub-identifier for QM. The “qmatrix_param( )” on line 29 is a functionspecifying quantization matrix parameters as illustrated by example inFIGS. 13 to 20.

FIG. 31 is an explanatory diagram illustrating an example of sliceheader syntax defined in accordance with the third technique. On line 5in FIG. 31, there is specified a reference PPS ID for referencingparameters included in the PPS from among the parameters to be set forthat slice. On line 8, there is specified a reference SUB_ALF ID forreferencing ALF-related parameters from among the parameters to be setfor that slice. On line 9, there is specified a reference SUB_SAO ID forreferencing SAO-related parameters from among the parameters to be setfor that slice. On line 10, there is specified a reference SUB_QM ID forreferencing QM-related parameters from among the parameters to be setfor that slice.

In the case of implementing the third technique, the parametergenerating section 130 of the syntax processing section 13 of the imageencoding device 10 attaches a new SUB_QM ID as a sub-identifier to anupdated group of quantization matrix parameters every time thequantization matrix parameters are updated. The inserting section 130then inserts the quantization matrix parameters with the attached SUB_QMID into the APS, together with other parameters. The parameter acquiringsection 160 of the syntax processing section 61 of the image decodingdevice 60 uses the reference SUB_QM ID designated in the slice header toacquire, from an APS, quantization matrix parameters to be set for eachslice.

According to the third technique, parameters are grouped inside the APSby using sub-identifiers, and the transmission of redundant parametersis not conducted for parameters in groups that do not require updating.For this reason, the encoding efficiency may be optimized. Also, sincethe classes of APSs do not increase even if the classes of parametersincrease, large numbers of NAL unit types are not expended as with thesecond technique discussed earlier. Consequently, the third techniquedoes not compromise the flexibility of future extensions.

In the example in FIGS. 29 to 31, parameters included in the APS aregrouped according to encoding tools relating to ALF, SAO, and QM.However, this is merely one example of grouping parameters. The APS mayalso include parameters relating to other encoding tools. For example,AIF-related parameters such as filter coefficients for an adaptiveinterpolation filter (AIF) are one example of parameters that may beincorporated into the APS. Hereinafter, various criteria for groupingparameters to be incorporated into the APS will be discussed withreference to FIG. 32.

The table illustrated in FIG. 32 lists “Parameter contents”, “Updatefrequency”, and “Data size” as features of respective parameters intypical encoding tools.

The adaptive loop filter (ALF) is a filter (typically a Wiener filter)that two-dimensionally filters a decoded image with filter coefficientsthat are adaptively determined so as to minimize the error between thedecoded image and the original image. ALF-related parameters includefilter coefficients to be applied to each block, and an on/off flag foreach coding unit (CU). The data size of ALF filter coefficients isextremely large compared to other classes of parameters. For thisreason, ALF-related parameters are ordinarily transmitted for high-rateI pictures, whereas the transmission of ALF-related parameters may beomitted for low-rate B pictures. This is because transmittingALF-related parameters with a large data size for low-rate pictures isinefficient from a gain perspective. In most cases, the ALF filtercoefficients vary for each picture. Since the filter coefficients dependon the image content, the likelihood of being able to reuse previouslyset filter coefficients is low.

The sample adaptive offset (SAO) is a tool that improves the imagequality of a decoded image by adding an adaptively determined offsetvalue to each pixel value in a decoded image. SAO-related parametersinclude offset patterns and offset values. The data size of SAO-relatedparameters is not as large as ALF-related parameters. SAO-relatedparameters likewise vary for each picture as a general rule. However,since SAO-related parameters have the property of not changing very mucheven if the image content changes slightly, there is a possibility ofbeing able to reuse previously set parameter values.

The quantization matrix (QM) is a matrix whose elements are quantizationscales used when quantizing transform coefficients transformed fromimage data by orthogonal transform. QM-related parameters, or in otherwords quantization matrix parameters, are as described in detail in thisspecification. The data size of QM-related parameters is larger thanSAO-related parameters. The quantization matrix is required for allpictures as a general rule, but does not necessarily required updatingfor every picture if the image content does not change greatly. For thisreason, the quantization matrix may be reused for the same picture types(such as I/P/B pictures), or for each GOP.

The adaptive interpolation filter (AIF) is a tool that adaptively variesthe filter coefficients of an interpolation filter used during motioncompensation for each sub-pixel position. AIF-related parameters includefilter coefficients for respective sub-pixel positions. The data size ofAIF-related parameters is small compared to the above three classes ofparameters. AIF-related parameters vary for each picture as a generalrule. However, since the same picture types tend to have similarinterpolation properties, AIF-related parameters may be reused for thesame picture types (such as I/P/B pictures).

On the basis of the above parameter qualities, the following threecriteria, for example, may be adopted for the purpose of groupingparameters included in the APS:

Criterion A) Grouping according to encoding tool

Criterion B) Grouping according to update frequency

Criterion C) Grouping according to likelihood of parameter reuse

Criterion A is a criterion that groups parameters according to theirrelated encoding tools. The parameter set structure illustrated byexample in FIGS. 29 to 31 are based on criterion A. Since the qualitiesof parameters are generally determined according to their relatedencoding tools, grouping parameters by encoding tool makes it possibleto make timely and efficient parameter updates according to the variousqualities of the parameters.

Criterion B is a criterion that groups parameters according to theirupdate frequency. As illustrated in FIG. 32, ALF-related parameters,SAO-related parameters, and AIF-related parameters all may be updatedevery picture as a general rule. Thus, these parameters can be groupedinto a single group while QM-related parameters are grouped into anothergroup, for example. In this case, there are fewer groups compared tocriterion A. As a result, there are also fewer sub-identifiers tospecify in the slice header, and the amount of codes of the slice headercan be reduced. Meanwhile, since the update frequencies of parametersbelonging to the same group resemble each other, the likelihood ofredundantly transmitting non-updated parameters in order to update otherparameters is kept low.

Criterion A is a criterion that groups parameters according to thelikelihood of parameter reuse. Although ALF-related parameters areunlikely to be reused, SAO-related parameters and AIF-related parametersare somewhat likely to be reused. With QM-related parameters, theparameters are highly likely to be reused over multiple pictures.Consequently, by grouping parameters according to their likelihood ofreuse in this way, the redundant transmission of reused parametersinside the APS can be avoided.

6-5. Exemplary Modification of Third Technique

With the third technique discussed above, the number of groups intowhich parameters are grouped inside the APS results in an equal numberof reference SUB IDs specified in the slice headers, as illustrated byexample in FIG. 31. The amount of codes required by the reference SUBIDs is approximately proportional to the product of the number of sliceheaders and the number of groups. A technique in accordance with theexemplary modification described below may also be implemented in orderto further reduce such a rate.

In the exemplary modification of the third technique, a combination IDassociated with a combination of sub-identifiers is defined inside theAPS or other parameter set. Parameters included inside the APS may thenbe referenced from a slice header via the combination ID. FIG. 33illustrates an example of an encoded stream configured according to suchan exemplary modification of the third technique.

Referring to FIG. 33, an SPS 831, a PPS 832, and an APS 833 are insertedat the start of a picture P0 positioned at the start of a sequence. ThePPS 832 is identified by the PPS ID “P0”. The APS 833 includesALF-related parameters, SAO-related parameters, and QM-relatedparameters. The ALF-related parameters are identified by a SUB_ALF ID“AA0”. The SAO-related parameters are identified by a SUB_SAO ID “AS0”.The QM-related parameters are identified by a SUB_QM ID “AQ0”.Additionally, the APS 833 includes a combination ID “C00”={AA0, AS0,AQ0} as a definition of a combination. A slice header 834 attached toslice data in the picture P0 includes the combination ID “C00”. Thismeans that the ALF-related parameters belonging to the SUB_ALF ID “AA0”,the SAO-related parameters belonging to the SUB_SAO ID “AS0”, and theQM-related parameters belonging to the SUB_QM ID “AQ0” that arerespectively associated with the combination ID “C00” are referenced inorder to decode that slice data.

An APS 835 is inserted into a picture P1 following the picture P0. TheAPS 835 includes ALF-related parameters and SAO-related parameters. TheALF-related parameters are identified by a SUB_ALF ID “AA1”. TheSAO-related parameters are identified by a SUB_SAO ID “AS1”. Since theQM-related parameters are not updated from the picture P0, QM-relatedparameters are not included in the APS 835. Additionally, the APS 835includes a combination ID “C01”={AA1, AS0, AQ0}, a combination ID“C02”={AA0, AS1, AQ0}, and a combination ID “C03”={AA1, AS1, AQ0} asdefinitions of combinations. A slice header 836 attached to slice datain the picture P1 includes the combination ID “C03”. This means that theALF-related parameters belonging to the SUB_ALF ID “AA1”, theSAO-related parameters belonging to the SUB_SAO ID “AS1”, and theQM-related parameters belonging to the SUB_QM ID “AQ0” that arerespectively associated with the combination ID “C03” are referenced inorder to decode that slice data.

An APS 837 is inserted into a picture P2 following the picture P1. TheAPS 837 includes ALF-related parameters. The ALF-related parameters areidentified by a SUB_ALF ID “AA2”. Since the SAO-related parameters andthe QM-related parameters are not updated from the picture P1,SAO-related parameters and QM-related parameters are not included in theAPS 837. Additionally, the APS 837 includes a combination ID “C04”={AA2,AS0, AQ0} and a combination ID “C05”={AA2, AS1, AQ0} as definitions ofcombinations. A slice header 838 attached to slice data in the pictureP2 includes the combination ID “C05”. This means that the ALF-relatedparameters belonging to the SUB_ALF ID “AA2”, the SAO-related parametersbelonging to the SUB_SAO ID “AS1”, and the QM-related parametersbelonging to the SUB_QM ID “AQ0” that are respectively associated withthe combination ID “C05” are referenced in order to decode that slicedata.

Note that in this exemplary modification, combination IDs may not bedefined for all combinations of sub-identifiers, such that combinationsIDs are defined only for the combinations of sub-identifiers actuallyreferenced in a slice header. Also, combinations of sub-identifiers maybe defined inside an APS different from the APS where the correspondingparameters are stored.

In the case of implementing this exemplary modification, the parametergenerating section 130 of the syntax processing section 13 of the imageencoding device 10 generates combination IDs as supplemental parameters,which are to be associated with combinations of sub-identifiers thatgroup various parameters, including quantization matrix parameters. Theinserting section 130 then inserts the combination IDs generated by theparameter generating section 130 into an APS or another parameter set.The parameter acquiring section 160 of the syntax processing section 61of the image decoding device 60 acquires the combination ID designatedin the slice header of each slice, and uses the sub-identifiersassociated with that combination ID to additionally acquire quantizationmatrix parameters inside the APS.

In this way, by using a combination ID associated with combinations ofsub-identifiers to reference parameters inside the APS, the amount ofcodes required to reference each parameter from the slice headers can bereduced.

7. EXAMPLE APPLICATION

The image encoding device 10 and the image decoding device 60 accordingto the embodiment described above may be applied to various electronicappliances such as a transmitter and a receiver for satellitebroadcasting, cable broadcasting such as cable TV, distribution on theInternet, distribution to client devices via cellular communication, andthe like, a recording device that records images onto a medium such asan optical disc, a magnetic disk, or flash memory, and a playback devicethat plays back images from such storage media. Four exampleapplications will be described below.

7-1. First Example Application

FIG. 34 is a block diagram illustrating an exemplary schematicconfiguration of a television adopting the embodiment described above. Atelevision 900 includes an antenna 901, a tuner 902, a demultiplexer903, a decoder 904, a video signal processing section 905, a displaysection 906, an audio signal processing section 907, a speaker 908, anexternal interface 909, a control section 910, a user interface 911, anda bus 912.

The tuner 902 extracts a signal of a desired channel from broadcastsignals received via the antenna 901, and demodulates the extractedsignal. Then, the tuner 902 outputs an encoded bit stream obtained bydemodulation to the demultiplexer 903. That is, the tuner 902 serves astransmission means of the television 900 for receiving an encoded streamin which an image is encoded.

The demultiplexer 903 separates a video stream and an audio stream of aprogram to be viewed from the encoded bit stream, and outputs theseparated streams to the decoder 904. Also, the demultiplexer 903extracts auxiliary data such as an electronic program guide (EPG) fromthe encoded bit stream, and supplies the extracted data to the controlsection 910. Additionally, the demultiplexer 903 may performdescrambling in the case where the encoded bit stream is scrambled.

The decoder 904 decodes the video stream and the audio stream input fromthe demultiplexer 903. Then, the decoder 904 outputs video datagenerated by the decoding process to the video signal processing section905. Also, the decoder 904 outputs the audio data generated by thedecoding process to the audio signal processing section 907.

The video signal processing section 905 plays back the video data inputfrom the decoder 904, and causes the display section 906 to display thevideo. The video signal processing section 905 may also cause thedisplay section 906 to display an application screen supplied via anetwork. Further, the video signal processing section 905 may performadditional processes such as noise removal, for example, on the videodata according to settings. Furthermore, the video signal processingsection 905 may generate graphical user interface (GUI) images such asmenus, buttons, or a cursor, for example, and superimpose the generatedimages onto an output image.

The display section 906 is driven by a drive signal supplied by thevideo signal processing section 905, and displays a video or an image ona video screen of a display device (such as a liquid crystal display, aplasma display, or an OLED display, for example).

The audio signal processing section 907 performs playback processes suchas D/A conversion and amplification on the audio data input from thedecoder 904, and outputs audio from the speaker 908. Also, the audiosignal processing section 907 may perform additional processes such asnoise removal on the audio data.

The external interface 909 is an interface for connecting the television900 to an external appliance or a network. For example, a video streamor an audio stream received via the external interface 909 may bedecoded by the decoder 904. That is, the external interface 909 alsoserves as transmission means of the televisions 900 for receiving anencoded stream in which an image is encoded.

The control section 910 includes a processor such as a centralprocessing unit (CPU), and memory such as random access memory (RAM),and read-only memory (ROM). The memory stores a program to be executedby the CPU, program data, EPG data, data acquired via a network, and thelike. The program stored in the memory is read and executed by the CPUwhen activating the television 900, for example. By executing theprogram, the CPU controls the operation of the television 900 accordingto an operation signal input from the user interface 911, for example.

The user interface 911 is connected to the control section 910. The userinterface 911 includes buttons and switches used by a user to operatethe television 900, and a remote control signal receiver, for example.The user interface 911 detects an operation by the user via thesestructural elements, generates an operation signal, and outputs thegenerated operation signal to the control section 910.

The bus 912 interconnects the tuner 902, the demultiplexer 903, thedecoder 904, the video signal processing section 905, the audio signalprocessing section 907, the external interface 909, and the controlsection 910.

In a television 900 configured in this way, the decoder 904 includes thefunctions of an image decoding device 60 according to the foregoingembodiments. Consequently, it is possible to moderate the decrease inencoding efficiency for video decoded by the television 900, or improvethe encoding efficiency.

7-2. Second Example Application

FIG. 35 is a block diagram illustrating an exemplary schematicconfiguration of a mobile phone adopting the embodiment described above.A mobile phone 920 includes an antenna 921, a communication section 922,an audio codec 923, a speaker 924, a microphone 925, a camera section926, an image processing section 927, a multiplexing/demultiplexing(mux/demux) section 928, a recording and playback section 929, a displaysection 930, a control section 931, an operable section 932, and a bus933.

The antenna 921 is connected to the communication section 922. Thespeaker 924 and the microphone 925 are connected to the audio codec 923.The operable section 932 is connected to the control section 931. Thebus 933 interconnects the communication section 922, the audio codec923, the camera section 926, the image processing section 927, themux/demux section 928, the recording and playback section 929, thedisplay 930, and the control section 931.

The mobile phone 920 performs operations such as transmitting andreceiving audio signals, transmitting and receiving emails or imagedata, taking images, and recording data in various operating modesincluding an audio communication mode, a data communication mode, animaging mode, and a videophone mode.

In the audio communication mode, an analog audio signal generated by themicrophone 925 is supplied to the audio codec 923. The audio codec 923converts the analog audio signal into audio data, and A/D converts andcompresses the converted audio data. Then, the audio codec 923 outputsthe compressed audio data to the communication section 922. Thecommunication section 922 encodes and modulates the audio data, andgenerates a transmit signal. Then, the communication section 922transmits the generated transmit signal to a base station (notillustrated) via the antenna 921. Also, the communication section 922amplifies a wireless signal received via the antenna 921 and convertsthe frequency of the wireless signal, and acquires a received signal.Then, the communication section 922 demodulates and decodes the receivedsignal and generates audio data, and outputs the generated audio data tothe audio codec 923. The audio codec 923 decompresses and D/A convertsthe audio data, and generates an analog audio signal. Then, the audiocodec 923 supplies the generated audio signal to the speaker 924 andcauses audio to be output.

Also, in the data communication mode, the control section 931 generatestext data that makes up an email, according to operations by a user viathe operable section 932, for example. Moreover, the control section 931causes the text to be displayed on the display section 930. Furthermore,the control section 931 generates email data according to transmitinstructions from the user via the operable section 932, and outputs thegenerated email data to the communication section 922. The communicationsection 922 encodes and modulates the email data, and generates atransmit signal. Then, the communication section 922 transmits thegenerated transmit signal to a base station (not illustrated) via theantenna 921. Also, the communication section 922 amplifies a wirelesssignal received via the antenna 921 and converts the frequency of thewireless signal, and acquires a received signal. Then, the communicationsection 922 demodulates and decodes the received signal, reconstructsthe email data, and outputs the reconstructed email data to the controlsection 931. The control section 931 causes the display section 930 todisplay the contents of the email, and also causes the email data to bestored in the storage medium of the recording and playback section 929.

The recording and playback section 929 includes an arbitrary readableand writable storage medium. For example, the storage medium may be abuilt-in storage medium such as RAM, or flash memory, or an externallymounted storage medium such as a hard disk, a magnetic disk, amagneto-optical disc, an optical disc, USB memory, or a memory card.

Furthermore, in the imaging mode, the camera section 926 takes an imageof a subject, generates image data, and outputs the generated image datato the image processing section 927, for example. The image processingsection 927 encodes the image data input from the camera section 926,and causes the encoded stream to be stored in the storage medium of therecording and playback section 929.

Furthermore, in the videophone mode, the mux/demux section 928multiplexes a video stream encoded by the image processing section 927and an audio stream input from the audio codec 923, and outputs themultiplexed stream to the communication section 922, for example. Thecommunication section 922 encodes and modulates the stream, andgenerates a transmit signal. Then, the communication section 922transmits the generated transmit signal to a base station (notillustrated) via the antenna 921. Also, the communication section 922amplifies a wireless signal received via the antenna 921 and convertsthe frequency of the wireless signal, and acquires a received signal.The transmit signal and received signal may include an encoded bitstream. Then, the communication section 922 demodulates and decodes thereceived signal, reconstructs the stream, and outputs the reconstructedstream to the mux/demux section 928. The mux/demux section 928 separatesa video stream and an audio stream from the input stream, and outputsthe video stream to the image processing section 927 and the audiostream to the audio codec 923. The image processing section 927 decodesthe video stream, and generates video data. The video data is suppliedto the display section 930, and a series of images is displayed by thedisplay section 930. The audio codec 923 decompresses and D/A convertsthe audio stream, and generates an analog audio signal. Then, the audiocodec 923 supplies the generated audio signal to the speaker 924 andcauses audio to be output.

In a mobile phone 920 configured in this way, the image processingsection 927 includes the functions of the image encoding device 10 andthe image decoding device 60 according to the foregoing embodiments.Consequently, it is possible to moderate the decrease in encodingefficiency for video encoded and decoded by the mobile phone 920, orimprove the encoding efficiency.

7-3. Third Example Application

FIG. 36 is a block diagram illustrating an exemplary schematicconfiguration of a recording and playback device adopting the embodimentdescribed above. A recording and playback device 940 encodes, andrecords onto a recording medium, the audio data and video data of areceived broadcast program, for example. The recording and playbackdevice 940 may also encode, and record onto the recording medium, audiodata and video data acquired from another device, for example.Furthermore, the recording and playback device 940 plays back datarecorded onto the recording medium via a monitor and speaker accordingto instructions from a user, for example. At such times, the recordingand playback device 940 decodes the audio data and the video data.

The recording and playback device 940 includes a tuner 941, an externalinterface 942, an encoder 943, a hard disk drive (HDD) 944, a disc drive945, a selector 946, a decoder 947, an on-screen display (OSD) 948, acontrol section 949, and a user interface 950.

The tuner 941 extracts a signal of a desired channel from broadcastsignals received via an antenna (not illustrated), and demodulates theextracted signal. Then, the tuner 941 outputs an encoded bit streamobtained by demodulation to the selector 946. That is, the tuner 941serves as transmission means of the recording and playback device 940.

The external interface 942 is an interface for connecting the recordingand playback device 940 to an external appliance or a network. Forexample, the external interface 942 may be an IEEE 1394 interface, anetwork interface, a USB interface, a flash memory interface, or thelike. For example, video data and audio data received by the externalinterface 942 are input into the encoder 943. That is, the externalinterface 942 serves as transmission means of the recording and playbackdevice 940.

In the case where the video data and the audio data input from theexternal interface 942 are not encoded, the encoder 943 encodes thevideo data and the audio data. Then, the encoder 943 outputs the encodedbit stream to the selector 946.

The HDD 944 records onto an internal hard disk an encoded bit stream,which is compressed content data such as video or audio, variousprograms, and other data. Also, the HDD 944 reads such data from thehard disk when playing back video and audio.

The disc drive 945 records or reads data with respect to an insertedrecording medium. The recording medium inserted into the disc drive 945may be a DVD disc (such as a DVD-Video, DVD-RAM, DVD-R, DVD-RW, DVD+, orDVD+RW disc), a Blu-ray (registered trademark) disc, or the like, forexample.

When recording video and audio, the selector 946 selects an encoded bitstream input from the tuner 941 or the encoder 943, and outputs theselected encoded bit stream to the HDD 944 or the disc drive 945. Also,when playing back video and audio, the selector 946 outputs an encodedbit stream input from the HDD 944 or the disc drive 945 to the decoder947.

The decoder 947 decodes the encoded bit stream, and generates video dataand audio data. Then, the decoder 947 outputs the generated video datato the OSD 948. Also, the decoder 904 outputs the generated audio datato an external speaker.

The OSD 948 plays back the video data input from the decoder 947, anddisplays video. Also, the OSD 948 may superimpose GUI images, such asmenus, buttons, or a cursor, for example, onto displayed video.

The control section 949 includes a processor such as a CPU, and memorysuch as RAM or ROM. The memory stores a program to be executed by theCPU, program data, and the like. A program stored in the memory is readand executed by the CPU when activating the recording and playbackdevice 940, for example. By executing the program, the CPU controls theoperation of the recording and playback device 940 according to anoperation signal input from the user interface 950, for example.

The user interface 950 is connected to the control section 949. The userinterface 950 includes buttons and switches used by a user to operatethe recording and playback device 940, and a remote control signalreceiver, for example. The user interface 950 detects an operation bythe user via these structural elements, generates an operation signal,and outputs the generated operation signal to the control section 949.

In a recording and playback device 940 configured in this way, theencoder 943 includes the functions of the image encoding device 10according to the foregoing embodiments. In addition, the decoder 947includes the functions of the image decoding device 60 according to theforegoing embodiments. Consequently, it is possible to moderate thedecrease in encoding efficiency for video encoded and decoded by therecording and playback device 940, or improve the encoding efficiency.

7-4. Fourth Example Application

FIG. 37 is a block diagram showing an example of a schematicconfiguration of an imaging device adopting the embodiment describedabove. An imaging device 960 takes an image of a subject, generates animage, encodes the image data, and records the image data onto arecording medium.

The imaging device 960 includes an optical block 961, an imaging section962, a signal processing section 963, an image processing section 964, adisplay section 965, an external interface 966, memory 967, a mediadrive 968, an OSD 969, a control section 970, a user interface 971, anda bus 972.

The optical block 961 is connected to the imaging section 962. Theimaging section 962 is connected to the signal processing section 963.The display section 965 is connected to the image processing section964. The user interface 971 is connected to the control section 970. Thebus 972 interconnects the image processing section 964, the externalinterface 966, the memory 967, the media drive 968, the OSD 969, and thecontrol section 970.

The optical block 961 includes a focus lens, an aperture stop mechanism,and the like. The optical block 961 forms an optical image of a subjecton the imaging surface of the imaging section 962. The imaging section962 includes an image sensor such as a CCD or CMOS sensor, andphotoelectrically converts the optical image formed on the imagingsurface into an image signal which is an electrical signal. Then, theimaging section 962 outputs the image signal to the signal processingsection 963.

The signal processing section 963 performs various camera signalprocesses such as knee correction, gamma correction, and colorcorrection on the image signal input from the imaging section 962. Thesignal processing section 963 outputs the processed image data to theimage processing section 964.

The image processing section 964 encodes the image data input from thesignal processing section 963, and generates encoded data. Then, theimage processing section 964 outputs the encoded data thus generated tothe external interface 966 or the media drive 968. Also, the imageprocessing section 964 decodes encoded data input from the externalinterface 966 or the media drive 968, and generates image data. Then,the image processing section 964 outputs the generated image data to thedisplay section 965. Also, the image processing section 964 may outputthe image data input from the signal processing section 963 to thedisplay section 965, and cause the image to be displayed. Furthermore,the image processing section 964 may superimpose display data acquiredfrom the OSD 969 onto an image to be output to the display section 965.

The OSD 969 generates GUI images such as menus, buttons, or a cursor,for example, and outputs the generated images to the image processingsection 964.

The external interface 966 is configured as an USB input/outputterminal, for example. The external interface 966 connects the imagingdevice 960 to a printer when printing an image, for example. Also, adrive is connected to the external interface 966 as necessary. Aremovable medium such as a magnetic disk or an optical disc, forexample, is inserted into the drive, and a program read from theremovable medium may be installed in the imaging device 960.Furthermore, the external interface 966 may be configured as a networkinterface to be connected to a network such as a LAN or the Internet.That is, the external interface 966 serves as transmission means of theimage capturing device 960.

A recording medium to be inserted into the media drive 968 may be anarbitrary readable and writable removable medium, such as a magneticdisk, a magneto-optical disc, an optical disc, or semiconductor memory,for example. Also, a recording medium may be permanently installed inthe media drive 968 to constitute a non-portable storage section such asan internal hard disk drive or a solid-state drive (SSD), for example.

The control section 970 includes a processor such as a CPU, and memorysuch as RAM or ROM. The memory stores a program to be executed by theCPU, program data, and the like. A program stored in the memory is readand executed by the CPU when activating the imaging device 960, forexample. By executing the program, the CPU controls the operation of theimaging device 960 according to an operation signal input from the userinterface 971, for example.

The user interface 971 is connected to the control section 970. The userinterface 971 includes buttons, switches and the like used by a user tooperate the imaging device 960, for example. The user interface 971detects an operation by the user via these structural elements,generates an operation signal, and outputs the generated operationsignal to the control section 970.

In an imaging device 960 configured in this way, the image processingsection 964 includes the functions of the image encoding device 10 andthe image decoding device 60 according to the foregoing embodiments.Consequently, it is possible to moderate the decrease in encodingefficiency for video encoded and decoded by the imaging device 960, orimprove the encoding efficiency.

8. CONCLUSION

The foregoing uses FIGS. 1 to 37 to describe an image encoding device 10and an image decoding device 60 according to an embodiment. According tothe embodiment, quantization matrix parameters defining quantizationmatrices used when quantizing and inversely quantizing an image areinserted into a quantization matrix parameter set (QMPS) that differsfrom the sequence parameter set and the picture parameter set. In sodoing, it becomes unnecessary both to encode parameters other thanquantization matrix parameters when updating a quantization matrix, andto encode quantization matrix parameters when updating parameters otherthan quantization matrix parameters. Consequently, the decrease inencoding efficiency accompanying the update of the quantization matrixis moderated, or the encoding efficiency is improved. Particularly, thereduction of the amount of codes with the techniques disclosed in thisspecification becomes even more effective in the case of quantizationmatrices with larger sizes, or in the case where a larger number ofquantization matrices are defined for each picture.

Also, according to the present embodiment, parameters that specifycopying a previously generated quantization matrix may be included inthe QMPS instead of directly defining a quantization matrix. In thiscase, the parameters that specify a quantization matrix itself (an arrayof differential data in DPCM format, for example) are omitted from theQMPS, and thus the amount of codes required to define a quantizationmatrix can be further reduced.

Also, according to the present embodiment, a QMPS ID is assigned to eachQMPS. Then, in copy mode, the QMPS ID of the QMPS defining the copysource quantization matrix may be designated as a source ID. Also, thesize and type of the copy source quantization matrix may be designatedas the copy source size and the copy source type. Consequently, aquantization matrix in an arbitrary QMPS from among quantizationmatrices in multiple QMPSs generated previously can be flexiblydesignated as a copy source quantization matrix. It is also possible tocopy and reuse a quantization matrix of a different size or type.

Also, according to the present embodiment, parameters that specifyresidual components of a quantization matrix to be copied may beincluded in a QMPS. Consequently, a quantization matrix that is notcompletely equal to a previously generated quantization matrix can stillbe generated anew at a low rate.

Also, in axis designation mode, instead of scanning all elements of aquantization matrix, only the values of the elements in the quantizationmatrix corresponding to three reference axes or the four corners of thequantization matrix may be included in the QMPS. Consequently, it islikewise possible to define a quantization matrix with a small amount ofcodes in this case.

Also, according to the present embodiment, the quantization matrices touse for each slice are set on the basis of quantization matrixparameters inside the QMPS identified by the QMPS ID designated in theslice header. Consequently, since quantization matrices can be flexiblyswitched for each slice, the optimal quantization matrix at each pointin time can be used to encode or decode video, even in the case wherethe image characteristics vary from moment to moment.

Note that this specification describes an example in which thequantization matrix parameter set multiplexed into the header of theencoded stream and transmitted from the encoding side to the decodingside. However, the technique of transmitting the quantization matrixparameter set is not limited to such an example. For example,information inside each parameter set may also be transmitted orrecorded as separate data associated with an encoded bit stream withoutbeing multiplexed into the encoded bit stream. Herein, the term“associated” means that images included in the bit stream (alsoencompassing partial images such as slices or blocks) and informationcorresponding to those images can be linked at the time of decoding. Inother words, information may also be transmitted on a separatetransmission channel from an image (or bit stream). Also, theinformation may be recorded to a separate recording medium (or aseparate recording area on the same recording medium) from the image (orbit stream). Furthermore, information and images (or bit streams) may beassociated with each other in arbitrary units such as multiple frames,single frames, or portions within frames, for example.

The foregoing thus describes preferred embodiments of the presentdisclosure in detail and with reference to the attached drawings.However, the technical scope of the present disclosure is not limited tosuch examples. It is clear to persons ordinarily skilled in thetechnical field to which the present disclosure belongs that variousmodifications or alterations may occur insofar as they are within thescope of the technical ideas stated in the claims, and it is to beunderstood that such modifications or alterations obviously belong tothe technical scope of the present disclosure.

Additionally, the present technology may also be configured as below.

(1)

An image processing device including:

an acquiring section configured to acquire quantization matrixparameters from an encoded stream in which the quantization matrixparameters defining a quantization matrix are set within a parameter setwhich is different from a sequence parameter set and a picture parameterset;

a setting section configured to set, based on the quantization matrixparameters acquired by the acquiring section, a quantization matrixwhich is used when inversely quantizing data decoded from the encodedstream; and

an inverse quantization section configured to inversely quantize thedata decoded from the encoded stream using the quantization matrix setby the setting section.

(2)

The image processing device according to (1), wherein

the parameter set containing the quantization matrix parameters is acommon parameter set within which other encoding parameters relating toencoding tools other than a quantization matrix can also be set, and

the acquiring section acquires the quantization matrix parameters whenthe quantization matrix parameters are set within the common parameterset.

(3)

The image processing device according to (2), wherein

the acquiring section determines whether the quantization matrixparameters are set within the common parameter set by referencing a flagincluded in the common parameter set.

(4)

The image processing device according to (2) or (3), wherein

the common parameter set is an adaptation parameter set.

(5)

The image processing device according to (4), wherein

in a case where a reference to a first adaptation parameter set isincluded in a second adaptation parameter set decoded after the firstadaptation parameter set, the setting section reuses a quantizationmatrix set based on the quantization matrix parameters acquired from thefirst adaptation parameter set as a quantization matrix corresponding tothe second adaptation parameter set.

(6)

The image processing device according to (5), wherein

in a case where a reference to a third adaptation parameter set isincluded within the third adaptation parameter set, the setting sectionsets a default quantization matrix as a quantization matrixcorresponding to the third adaptation parameter set.

(7)

The image processing device according to (1), wherein

in a case where a copy parameter instructing to copy a firstquantization matrix for a first parameter set is included in a secondparameter set, the setting section sets a second quantization matrix bycopying the first quantization matrix.

(8)

The image processing device according to (7), wherein

each parameter set that includes the quantization matrix parameters hasan identifier that identifies each parameter set, and

the copy parameter includes an identifier of a parameter set of a copysource.

(9)

The image processing device according to (8), wherein

each parameter set includes quantization matrix parameters thatrespectively define a plurality of classes of quantization matrices, and

the copy parameter includes a class parameter designating a class of thefirst quantization matrix.

(10)

The image processing device according to (8), wherein

in a case where the identifier of the parameter set of the copy sourceincluded in the third parameter set is equal to an identifier of thethird parameter set, the setting section sets a default quantizationmatrix as a third quantization matrix for the third parameter set.

(11)

The image processing device according to (7), wherein

in a case where a size of the second quantization matrix is larger thana size of the first quantization matrix, the setting section sets thesecond quantization matrix by interpolating elements of the copied firstquantization matrix.

(12)

The image processing device according to (7), wherein

in a case where a size of the second quantization matrix is smaller thana size of the first quantization matrix, the setting section sets thesecond quantization matrix by decimating elements of the copied firstquantization matrix.

(13)

The image processing device according to (7), wherein

in a case where a residual designation parameter designating residualcomponents of a quantization matrix which is copied is included in thesecond parameter set, the setting section sets the second quantizationmatrix by adding the residual components to the copied firstquantization matrix.

(14)

The image processing device according to (1), wherein

each parameter set that includes the quantization matrix parameters hasa parameter set identifier that identifies each parameter set, and

the inverse quantization section uses, for each slice, a quantizationmatrix set by the setting section based on quantization matrixparameters included in a parameter set identified by the parameter setidentifier designated in a slice header.

(15)

The image processing device according to any one of (7) to (14), wherein

the parameter set containing the quantization matrix parameters furtherincludes other encoding parameters relating to encoding tools other thana quantization matrix.

(16)

The image processing device according to (15), wherein

the quantization matrix parameters and the other encoding parameters aregrouped by a sub-identifier defined separately from a parameteridentifier that identifies each parameter set, and

the acquiring section acquires the quantization matrix parameters usingthe sub-identifier.

(17)

The image processing device according to (16), wherein

a combination identifier associated with a combination of a plurality ofthe sub-identifiers is defined in the parameter set or another parameterset, and

the acquiring section acquires the combination identifier designated ina slice header of each slice, and acquires the quantization matrixparameters using the sub-identifiers associated with the acquiredcombination identifier.

(18)

An image processing method including:

acquiring quantization matrix parameters from an encoded stream in whichthe quantization matrix parameters defining a quantization matrix areset within a parameter set which is different from a sequence parameterset and a picture parameter set;

setting, based on the acquired quantization matrix parameters, aquantization matrix which is used when inversely quantizing data decodedfrom the encoded stream; and

inversely quantizing the data decoded from the encoded stream using theset quantization matrix.

(19)

An image processing device including:

a quantization section configured to quantize data using a quantizationmatrix;

a setting section configured to set quantization matrix parameters thatdefine a quantization matrix to be used when the quantization sectionquantizes the data; and

an encoding section configured to encode the quantization matrixparameters set by the setting section within a parameter set which isdifferent from a sequence parameter set and a picture parameter set.

(20)

An image processing method including:

quantizing data using a quantization matrix;

setting quantization matrix parameters that define the quantizationmatrix to be used when quantizing the data; and

encoding the set quantization matrix parameters within a parameter setwhich is different from a sequence parameter set and a picture parameterset.

REFERENCE SIGNS LIST

-   10 Image processing device (image encoding device)-   16 Quantization section-   120 Parameter generating section-   130 Inserting section-   60 Image processing device (image decoding device)-   63 Inverse quantization section-   160 Parameter acquiring section-   170 Setting section

The invention claimed is:
 1. An image processing device comprising:circuitry configured to: determine whether or not a current quantizationmatrix is to be generated by copying a source quantization matrix,wherein the current quantization matrix comprises a secondidentification value; in a case where the current quantization matrix isto be generated by copying the source quantization matrix, encode afirst information into a bit stream, wherein the first informationindicates the current quantization matrix is to be generated by copyingthe source quantization matrix; determine whether a default matrix is tobe used as the source quantization matrix;  in a case where the defaultmatrix is to be used as the source quantization matrix, set a firstidentification value equal to the second identification value;  in acase where the default matrix is not to be used as the sourcequantization matrix, set the first identification value not equal to thesecond identification value such that the source quantization matrix canbe identified based at least in part on the first identification value,and based on one or more or none of: a matrix size and a matrix type;encode, into the bit stream, a second information which is indicative ofthe first identification value; and encode an image using the currentquantization matrix to form part of the bit stream.
 2. The imageprocessing device according to claim 1, wherein the matrix type includesat least one combination of a prediction mode and a color component. 3.The image processing device according to claim 2, wherein the at leastone combination of the prediction mode and the color component includesintra prediction for luma component, intra prediction for chromacomponent, inter prediction for luma component, and inter prediction forchroma component.
 4. The image processing device according to claim 2,wherein the at least one combination of the prediction mode and thecolor component includes intra prediction for Y component, intraprediction for Cb component, intra prediction for Cr component, interprediction for Y component, inter prediction for Cb component, and interprediction for Cr component.
 5. The image processing device according toclaim 1, wherein in a case where the current quantization matrix is notto be generated by copying the source quantization matrix: encode, intothe bit stream, the current quantization matrix; and encode the imageusing the current quantization matrix to form part of the bit stream. 6.The image processing device according to claim 1, wherein the bit streamis transmitted.
 7. The image processing device according to claim 5,wherein the bit stream is transmitted.
 8. An image processing methodcomprising the steps of: determining whether or not a currentquantization matrix is to be generated by copying a source quantizationmatrix, wherein the current quantization matrix comprises a secondidentification value; in the case where the current quantization matrixis to be generated by copying the source quantization matrix, encoding afirst information into a bit stream, wherein the first informationindicates the current quantization matrix is to be generated by copyingthe source quantization matrix; determining whether a default matrix isto be used as the source quantization matrix; in a case where thedefault matrix is to be used as the source quantization matrix, settinga first identification value equal to the second identification value;in a case where the default matrix is not to be used as the sourcequantization matrix, setting a first identification value not equal tothe second identification value such that the source quantization matrixcan be identified based at least in part on the first identificationvalue, and based on one or more or none of: a matrix size and a matrixtype; encoding, into the bit stream, a second information which isindicative of the first identification value; and, encoding an imageusing the current quantization matrix to form part of the bit stream. 9.The image processing method according to claim 8, wherein the matrixtype includes at least one combination of a prediction mode and a colorcomponent.
 10. The image processing method according to claim 9, whereinthe at least one combination of the prediction mode and the colorcomponent includes intra prediction for luma component, intra predictionfor chroma component, inter prediction for luma component, and interprediction for chroma component.
 11. The image processing methodaccording to claim 9, wherein the at least one combination of theprediction mode and the color component includes intra prediction for Ycomponent, intra prediction for Cb component, intra prediction for Crcomponent, inter prediction for Y component, inter prediction for Cbcomponent, and inter prediction for Cr component.
 12. The imageprocessing method according to claim 8, wherein a case where the currentquantization matrix is not to be generated by copying the sourcequantization matrix: encoding, into the bit stream, the currentquantization matrix; and, encoding the image using the currentquantization matrix to form part of the bit stream.
 13. The imageprocessing device according to claim 8, wherein the bit stream istransmitted.
 14. The image processing device according to claim 12,wherein the bit stream is transmitted.